Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/latest/pull_cdc_integration_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm Agent pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-190bl-239nt --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "18c1b448f14ace3f8ae1f3988373c3c9bb1e6fb1" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-190bl" name: "pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git rev-list --no-walk b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} _=/usr/bin/printenv TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-190bl HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=d88e55cc-3ac1-4e4a-879a-3e8fb9f49c0d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Debug info JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=3 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-190bl pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-ff467ea3dec646dcb34a434b6d5ddeaf65f47c579492826166e1422a0f0cf5b0 + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.23.3' GODEBUG='' GOTELEMETRY='local' GOTELEMETRYDIR='/home/jenkins/.config/go/telemetry' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1521897757=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm bash' debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-test-3828-190bl-239nt-xhqwm bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51573 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: d6d4e302b7d86003 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 27 IN A 20.27.177.113 github.com. 27 IN A 20.205.243.166 ;; Query time: 1 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 04 07:24:05 UTC 2025 ;; MSG SIZE rcvd: 103 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [.github/workflows/check_and_build.yaml, .github/workflows/dataflow_engine_chaos.yaml, .github/workflows/dm_binlog_999999.yaml, .github/workflows/dm_chaos.yaml, .github/workflows/dm_mariadb_master_down_and_up.yaml, .github/workflows/dm_upstream_switch.yaml] [Pipeline] echo diff file not matched: .github/workflows/check_and_build.yaml [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-4c631d5-a306f48) 192337920 bytes in 0.70 secs (274980830 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.43.5 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/.git/ .git HEAD is now at a306f4894 dm: upgrade go-mysql to v1.12.0 POST git-upload-pack (656 bytes) POST git-upload-pack (973 bytes) From https://github.com/pingcap/tiflow = [up to date] master -> origin/master * [new ref] refs/pull/12120/head -> origin/pr/12120/head Previous HEAD position was a306f4894 dm: upgrade go-mysql to v1.12.0 HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) ๐Ÿšง Checkouting to base SHA:4c631d5951854ac91765b729fe3849d8e8d5ed8a... HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 4c631d5951854ac91765b729fe3849d8e8d5ed8a 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) 9ed3d76c5 consumer(ticdc): adjust consumer log level (#12092) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 4c631d5951854ac91765b729fe3849d8e8d5ed8a ... Updating 4c631d595..e8bfbe4ba Fast-forward .github/workflows/check_and_build.yaml | 2 +- .github/workflows/dataflow_engine_chaos.yaml | 2 +- .github/workflows/dm_binlog_999999.yaml | 4 ++-- .github/workflows/dm_chaos.yaml | 4 ++-- .github/workflows/dm_mariadb_master_down_and_up.yaml | 4 ++-- .github/workflows/dm_upstream_switch.yaml | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) ๐Ÿงพ Pre-merged result: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 e8bfbe4ba CI: Upgrade to actions/cache@v4 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-4c631d5-e8bfbe4 already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/third_party_download [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] echo computeBranchFromPR component: tidb, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: pd, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tikv, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tiflash, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] retry [Pipeline] { [Pipeline] sh + export TIDB_BRANCH=master + TIDB_BRANCH=master + export PD_BRANCH=master + PD_BRANCH=master + export TIKV_BRANCH=master + TIKV_BRANCH=master + export TIFLASH_BRANCH=master + TIFLASH_BRANCH=master + cd ../tiflow + ./scripts/download-integration-test-binaries.sh master Downloading binaries... >>> Downloading tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz 2025-04-04 15:24:31 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz [620275338/620275338] -> "tmp/tidb-server.tar.gz" [1] >>> Downloading pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz 2025-04-04 15:24:47 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz [228444265/228444265] -> "tmp/pd-server.tar.gz" [1] >>> Downloading tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz 2025-04-04 15:25:08 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz [817400808/817400808] -> "tmp/tikv-server.tar.gz" [1] >>> Downloading tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz 2025-04-04 15:25:33 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz [553235170/553235170] -> "tmp/tiflash.tar.gz" [1] >>> Downloading minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2025-04-04 15:25:39 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> Downloading etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2025-04-04 15:25:40 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> Downloading schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2025-04-04 15:25:49 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] >>> Downloading go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> Downloading jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] Download SUCCESS + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + make check_third_party_binary go: downloading go1.23.4 (linux/amd64) /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-region-bench ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/pd-ut ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ../tiflow/bin/xprog ./bin/ + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + ./bin/tidb-server -V Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + ./bin/tikv-server -V TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:59" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/sync v0.10.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading github.com/stretchr/testify v1.10.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading golang.org/x/text v0.21.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading k8s.io/apimachinery v0.29.11 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:29:02" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer go: downloading github.com/confluentinc/confluent-kafka-go/v2 v2.3.0 go: downloading github.com/DataDog/zstd v1.5.5 + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/sync_diff_inspector ./sync_diff_inspector go: downloading github.com/olekukonko/tablewriter v0.0.5 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20241224111350-ad2e4f976c7c go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading gorm.io/driver/mysql v1.5.7 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/sqlite v1.27.0 go: downloading modernc.org/libc v1.37.1 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.4G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:37 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:37 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:37 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:32 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:32 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:31 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:32 oauth2-server -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:32 sync_diff_inspector + ./bin/cdc version Release Version: v9.0.0-beta.1-10-ge8bfbe4ba Git Commit Hash: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 Git Branch: HEAD UTC Build Time: 2025-04-04 07:31:44 Go Version: go1.23.4 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-4c631d5-e8bfbe4) 1485414400 bytes in 29.02 secs (51186633 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-region-bench ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/pd-ut ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ../third_party_download/bin/xprog ./bin/ + ls -alh ./bin total 3.3G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:38 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:38 bin -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:37 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:37 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:32 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:32 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:31 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:38 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 4 15:38 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:38 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:38 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:38 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:38 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Apr 4 15:38 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 4 15:38 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Apr 4 15:38 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:38 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 4 15:38 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 4 15:38 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 4 15:38 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 4 15:38 minio -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:32 oauth2-server -rwxr-xr-x 1 jenkins jenkins 41M Apr 4 15:38 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:38 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 4 15:38 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:38 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:38 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 4 15:38 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:38 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 4 15:38 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:38 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:38 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:38 stores-dump -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:32 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 241M Apr 4 15:38 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 4 15:38 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 4 15:38 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 4 15:38 xprog [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 74.03 secs (54442211 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G18') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G19') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G20') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G21') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G18') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G19') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G20') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G21') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs Agent pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-s8jj0-xfl8k --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "cbc70d756c86765011a134a7429c503b6154c8ec" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-s8jj0" name: "pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Agent pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-csk1g-gh5zq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6934bbef8f267c4b261e4d840d01430f442e92bb" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-csk1g" name: "pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Agent pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-d3bms-87pgm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f15a4acc86ca90055922ce21d77b2bfad6b8a96e" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-d3bms" name: "pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@528d34cc; decorates RemoteLauncher[hudson.remoting.Channel@7439fe9b:JNLP4-connect connection from 10.233.127.176/10.233.127.176:55282] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@14910b62; decorates RemoteLauncher[hudson.remoting.Channel@2f68be91:JNLP4-connect connection from 10.233.75.215/10.233.75.215:43320] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Agent pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-0vmr4-q18gq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8a2087b9cdee2fe36016ac7f76ce40079f2b7b2b" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-0vmr4" name: "pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] checkout Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@44302036; decorates RemoteLauncher[hudson.remoting.Channel@7129c4e0:JNLP4-connect connection from 10.233.97.170/10.233.97.170:38278] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6683c59f; decorates RemoteLauncher[hudson.remoting.Channel@1b06e62d:JNLP4-connect connection from 10.233.93.136/10.233.93.136:47440] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-9m9ct-6m5th --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "27a7ee4838478f0c03e04aba9ef4d626b2c3905c" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-9m9ct" name: "pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] checkout Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git The recommended git tool is: git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@46664e9a; decorates RemoteLauncher[hudson.remoting.Channel@911455a:JNLP4-connect connection from 10.233.127.73/10.233.127.73:54362] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-xh4jw-bv9zd --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "a66d96cc06d754577d92a582164d83ca8f63c235" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-xh4jw" name: "pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-l1r4j-cwb9x --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f297d6fc7c5f3ce44c584670ece0ad110c66e5f7" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-l1r4j" name: "pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-t3v13-m4jvc --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "bcb460b872444dd4c311e59752445cfe6f2ffbbf" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-t3v13" name: "pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-23b4k-7ks82 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "068c6ca682ff4456ca9b7eaf727f3ba9ff107374" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-23b4k" name: "pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Avoid second fetch Running on pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-zl944-bffst --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "53921dba93c8bd73eef533b46421d01e27f8ee12" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-zl944" name: "pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-7dz70-dw7fn --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f69a8869de7e2387cb85e7df84e9df08da8c3a34" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-7dz70" name: "pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.56 secs (470782939 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G00 [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node Run cases: down_db_require_secure_transport bdr_mode capture_suicide_while_balance_table syncpoint syncpoint_check_ts server_config_compatibility changefeed_dup_error_restart safe_mode kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update csv_storage_update_pk_clustered csv_storage_update_pk_nonclustered GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-s8jj0 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=8bd241ed-1b13-4048-a813-9660dfe0b240 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G00 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-s8jj0 pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/down_db_require_secure_transport/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] { The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2e149870; decorates RemoteLauncher[hudson.remoting.Channel@5d20b014:JNLP4-connect connection from 10.233.97.39/10.233.97.39:52648] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@36dbeb53; decorates RemoteLauncher[hudson.remoting.Channel@7256e5d8:JNLP4-connect connection from 10.233.71.157/10.233.71.157:55852] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@570e395b; decorates RemoteLauncher[hudson.remoting.Channel@4404b99d:JNLP4-connect connection from 10.233.93.213/10.233.93.213:51256] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2a937267; decorates RemoteLauncher[hudson.remoting.Channel@1b682784:JNLP4-connect connection from 10.233.75.171/10.233.75.171:41324] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4566f165; decorates RemoteLauncher[hudson.remoting.Channel@79b530c4:JNLP4-connect connection from 10.233.73.175/10.233.73.175:38512] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@286eabe; decorates RemoteLauncher[hudson.remoting.Channel@1344ef7f:JNLP4-connect connection from 10.233.127.212/10.233.127.212:49584] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-htn1z-434rm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c8cc850dbbf2b2624884412ea5c23ace155dd737" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-htn1z" name: "pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" start tidb cluster in /tmp/tidb_cdc_test/down_db_require_secure_transport Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Running on pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Agent pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-1hc17-b4qpf --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "79b749a10f352c6f4ce8c9816e4f69a6f85e1c89" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-1hc17" name: "pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-r1g2f-25j5p --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "a5e5964306f98c1cc0f1da4e0d8c19bd42990416" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-r1g2f" name: "pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-7b293-7dhtl --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "5459d0484f27c89b1cc47102d2e5d5ec47e05ac5" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-7b293" name: "pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Running on pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-k1gsf-t0xw1 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "1e8d260247e4520e7b5f1deb8b5c76cb2396c64a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-k1gsf" name: "pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Verifying downstream PD is started... Agent pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-9jjm4-656tq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c7146b12c62edadc9e5597963b8480503e836d0d" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-9jjm4" name: "pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Running on pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-1z1c1-lmh5n --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "28c24ab145c7c19cb92db2b09f6f852e5b41023a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-1z1c1" name: "pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-9ffrv-vdlp8 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "69dd90b8c355a6c6d4a603b377efc32e2b7b63f9" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-9ffrv" name: "pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-xb2bm-8cz4s --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "4cfa82709b192d611f05400e22ffc65917033edd" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-xb2bm" name: "pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-9cfmv-g5xz7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ead7b3f69d5583d1165a49381c759739f60073aa" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-9cfmv" name: "pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3828-87zdl-ttvzb --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "94eeacffc23c2e44cb43ac4e4d19e8d266d9e90d" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3828-87zdl" name: "pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) config file /tmp/tidb_cdc_test/down_db_require_secure_transport/tidb-config-1743752390533253726.toml contained invalid configuration options: security.socket; check TiDB manual to make sure this option has not been deprecated and removed from your TiDB version if the option does not appear to be a typo ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.84 secs (456128467 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G01 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359473813618377 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7bf680002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:1285, start at 2025-04-04 15:39:52.667063604 +0800 CST m=+2.205920834 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.666 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.666 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359473813618377 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7bf680002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:1285, start at 2025-04-04 15:39:52.667063604 +0800 CST m=+2.205920834 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.666 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.666 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359476624079127 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7c240000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:1350, start at 2025-04-04 15:39:52.889038854 +0800 CST m=+2.343887623 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.897 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.897 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth debezium_basic kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-csk1g HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=3b2d77aa-690a-4017-b33e-3fdf4434a06e GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G01 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd pingcap_tiflow_pull_cdc_integration_test_3828-csk1g PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Collecting requests==2.26.0 Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Collecting certifi>=2017.4.17 (from requests==2.26.0) Downloading https://files.pythonhosted.org/packages/38/fc/bce832fd4fd99766c04d1ee0eead6b0ec6486fb100ae5e74c1d91292b982/certifi-2025.1.31-py3-none-any.whl (166kB) Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Installing collected packages: certifi, requests Found existing installation: certifi 2024.8.30 Uninstalling certifi-2024.8.30: Successfully uninstalled certifi-2024.8.30 Found existing installation: requests 2.27.1 Uninstalling requests-2.27.1: Successfully uninstalled requests-2.27.1 Successfully installed certifi-2025.1.31 requests-2.26.0 [Fri Apr 4 15:39:57 CST 2025] <<<<<< START cdc server in down_db_require_secure_transport case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.down_db_require_secure_transport.26592661.out server --log-file /tmp/tidb_cdc_test/down_db_require_secure_transport/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/down_db_require_secure_transport/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:00 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46539559-f08f-42a1-b571-727bf1906482 {"id":"46539559-f08f-42a1-b571-727bf1906482","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd44bda 46539559-f08f-42a1-b571-727bf1906482 /tidb/cdc/default/default/upstream/7489359473813618377 {"id":7489359473813618377,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46539559-f08f-42a1-b571-727bf1906482 {"id":"46539559-f08f-42a1-b571-727bf1906482","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd44bda 46539559-f08f-42a1-b571-727bf1906482 /tidb/cdc/default/default/upstream/7489359473813618377 {"id":7489359473813618377,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46539559-f08f-42a1-b571-727bf1906482 {"id":"46539559-f08f-42a1-b571-727bf1906482","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd44bda 46539559-f08f-42a1-b571-727bf1906482 /tidb/cdc/default/default/upstream/7489359473813618377 {"id":7489359473813618377,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.down_db_require_secure_transport.cli.2722.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?ssl-ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem&ssl-key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem&ssl-cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem' --changefeed-id=down-db-require-secure-transport Create changefeed successfully! ID: down-db-require-secure-transport Info: {"upstream_id":7489359473813618377,"namespace":"default","id":"down-db-require-secure-transport","sink_uri":"mysql://root@127.0.0.1:3306/?ssl-ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem\u0026ssl-key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem\u0026ssl-cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem","create_time":"2025-04-04T15:40:02.57450342+08:00","start_ts":457114229778939909,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114229778939909,"checkpoint_ts":457114229778939909,"checkpoint_time":"2025-04-04 15:40:02.416"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/http_api Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.91 secs (509843321 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table down_db_require_secure_transport.finishmark not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table down_db_require_secure_transport.finishmark exists [Fri Apr 4 15:40:09 CST 2025] <<<<<< run test case down_db_require_secure_transport success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359547990597012 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d1c4003e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:1447, start at 2025-04-04 15:40:10.25656235 +0800 CST m=+2.775758540 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359547990597012 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d1c4003e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:1447, start at 2025-04-04 15:40:10.25656235 +0800 CST m=+2.775758540 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359550217877430 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d0380006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:1520, start at 2025-04-04 15:40:10.134787334 +0800 CST m=+2.566051990 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.143 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 6.90 secs (583730924 bytes/sec) [Pipeline] { [Pipeline] cache [Fri Apr 4 15:40:13 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.27542756.out server --log-file /tmp/tidb_cdc_test/http_api/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:16 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/default/upstream/7489359547990597012 + grep -q 'failed to get info:' {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' "is-owner": true, run task successfully owner pid: 2759 owner id 687f4c7b-1d86-4208-8553-e41ad7fc119a default [Fri Apr 4 15:40:19 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.28732875.out server --log-file /tmp/tidb_cdc_test/http_api/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:22 GMT < Content-Length: 1305 < Content-Type: text/plain; charset=utf-8 < { [1305 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/687f4c7b-1d86-4208-8553-e41ad7fc119a {"id":"687f4c7b-1d86-4208-8553-e41ad7fc119a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ae3 687f4c7b-1d86-4208-8553-e41ad7fc119a /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "687f4c7b-1d86-4208-8553-e41ad7fc119a default" | grep id Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.41 secs (479442694 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh start tidb cluster in /tmp/tidb_cdc_test/bdr_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... "id": "e5b2cf76-ec42-479f-8c58-0fff42a4aa91", run task successfully + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G02 [Pipeline] sh Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-d3bms HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=6b61915f-fedd-498f-9e7e-d0d42f8a85ed GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G02 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6 pingcap_tiflow_pull_cdc_integration_test_3828-d3bms PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G03 [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { Run cases: row_format drop_many_tables processor_stop_delay partition_table ddl_with_exists GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-0vmr4 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=2bb7d4c2-e59f-4d70-be5b-aa1cfa01002e GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G03 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-0vmr4 pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G04 [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-9m9ct HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=3fd08e27-10f8-4d19-9537-12bef7696b8a GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G04 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-9m9ct pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { capture_id: e5b2cf76-ec42-479f-8c58-0fff42a4aa91 pass test: check health pass test: get status [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { pass test: create changefeed [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@67bf7a78; decorates RemoteLauncher[hudson.remoting.Channel@43eea15c:JNLP4-connect connection from 10.233.97.24/10.233.97.24:45686] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@73c756ef; decorates RemoteLauncher[hudson.remoting.Channel@552cb7b3:JNLP4-connect connection from 10.233.74.246/10.233.74.246:47680] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3c2b1552; decorates RemoteLauncher[hudson.remoting.Channel@68618f4c:JNLP4-connect connection from 10.233.107.215/10.233.107.215:54140] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6820c4c; decorates RemoteLauncher[hudson.remoting.Channel@1a0f398b:JNLP4-connect connection from 10.233.77.180/10.233.77.180:34284] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@73e017b; decorates RemoteLauncher[hudson.remoting.Channel@65bad937:JNLP4-connect connection from 10.233.88.34/10.233.88.34:49670] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@56bb7546; decorates RemoteLauncher[hudson.remoting.Channel@2a9c65df:JNLP4-connect connection from 10.233.90.13/10.233.90.13:53780] will be ignored (a typical symptom is the Git executable not being run inside a designated container) [Pipeline] withCredentials Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@42705263; decorates RemoteLauncher[hudson.remoting.Channel@6b2f33f1:JNLP4-connect connection from 10.233.93.110/10.233.93.110:46788] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@64975c78; decorates RemoteLauncher[hudson.remoting.Channel@4fbd3edf:JNLP4-connect connection from 10.233.127.98/10.233.127.98:51750] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@a2a1a4d; decorates RemoteLauncher[hudson.remoting.Channel@2bccf653:JNLP4-connect connection from 10.233.75.249/10.233.75.249:49280] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@73d810c7; decorates RemoteLauncher[hudson.remoting.Channel@20fd206b:JNLP4-connect connection from 10.233.66.186/10.233.66.186:44778] will be ignored (a typical symptom is the Git executable not being run inside a designated container) No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@c80d66b; decorates RemoteLauncher[hudson.remoting.Channel@5b66728c:JNLP4-connect connection from 10.233.73.229/10.233.73.229:36124] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] cache Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) table test.simple1 not exists for 1-th check, retry later Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Verifying downstream PD is started... > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.simple1 exists pass test: list changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: get changefeed Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: pause changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: update changefeed wait process 2759 exit for 1-th time... wait process 2759 exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 2759 exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.40 secs (544646173 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2759) - No such process wait process 2759 exit for 4-th time... process 2759 already exit cdc.test cli capture list --server http://127.0.0.1:8301 2>&1 |grep e5b2cf76-ec42-479f-8c58-0fff42a4aa91 -A1 | grep '"is-owner": true' "is-owner": true, run task successfully [Fri Apr 4 15:40:39 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.31313133.out server --log-file /tmp/tidb_cdc_test/http_api/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359667513337815 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa71080015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:1208, start at 2025-04-04 15:40:36.849774644 +0800 CST m=+2.227255484 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:36.858 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:36.902 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:36.902 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359667513337815 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa71080015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:1208, start at 2025-04-04 15:40:36.849774644 +0800 CST m=+2.227255484 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:36.858 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:36.902 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:36.902 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359655541398866 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa73780004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:1285, start at 2025-04-04 15:40:36.963449706 +0800 CST m=+2.241163850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:36.971 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.007 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.007 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359661149374804 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7c700012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:1280, start at 2025-04-04 15:40:37.558535131 +0800 CST m=+2.512151424 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.569 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.532 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.532 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359661149374804 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7c700012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:1280, start at 2025-04-04 15:40:37.558535131 +0800 CST m=+2.512151424 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.569 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.532 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.532 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359665842127115 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7e000005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:1357, start at 2025-04-04 15:40:37.643409615 +0800 CST m=+2.497244752 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.632 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.632 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359661716040254 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7ac40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:1353, start at 2025-04-04 15:40:37.42617457 +0800 CST m=+2.211726493 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.433 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.425 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.425 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359661716040254 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7ac40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:1353, start at 2025-04-04 15:40:37.42617457 +0800 CST m=+2.211726493 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.433 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.425 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.425 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359668161538766 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa7e50000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:1419, start at 2025-04-04 15:40:37.667955261 +0800 CST m=+2.389488565 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:37.675 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:37.652 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:37.652 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:40:41 CST 2025] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.26732675.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server1 --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2647.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2647.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:40:26.613576086 +0800 CST StartTs:457114236109717518 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed19e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236109717521} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240041877512, checkpointTs: 457114240041877512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing changefeedID: default/changefeed-test1 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:26.406997872 +0800 CST StartTs:457114236043919370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed0240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236057026569} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2222a11-decb-457c-86eb-bc48b85384eb {"id":"b2222a11-decb-457c-86eb-bc48b85384eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752439} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ba4 b2222a11-decb-457c-86eb-bc48b85384eb /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:26.406997872+08:00","start-ts":457114236043919370,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236057026569} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.514495707+08:00","start-ts":457114236083503115,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238049320972} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.613576086+08:00","start-ts":457114236109717518,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236109717521} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114236581314565,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:40:26.613576086 +0800 CST StartTs:457114236109717518 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed19e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236109717521} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240041877512, checkpointTs: 457114240041877512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing changefeedID: default/changefeed-test1 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:26.406997872 +0800 CST StartTs:457114236043919370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed0240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236057026569} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2222a11-decb-457c-86eb-bc48b85384eb {"id":"b2222a11-decb-457c-86eb-bc48b85384eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752439} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ba4 b2222a11-decb-457c-86eb-bc48b85384eb /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:26.406997872+08:00","start-ts":457114236043919370,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236057026569} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.514495707+08:00","start-ts":457114236083503115,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238049320972} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.613576086+08:00","start-ts":457114236109717518,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"bi+ grep -q 'failed to get info:' nary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236109717521} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114236581314565,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:40:26.613576086 +0800 CST StartTs:457114236109717518 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed19e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236109717521} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240041877512, checkpointTs: 457114240041877512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing changefeedID: default/changefeed-test1 {UpstreamID:7489359547990597012 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:26.406997872 +0800 CST StartTs:457114236043919370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ed0240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114236057026569} {CheckpointTs:457114236581314565 MinTableBarrierTs:457114240041877512 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114236581314565, checkpointTs: 457114236581314565, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2222a11-decb-457c-86eb-bc48b85384eb {"id":"b2222a11-decb-457c-86eb-bc48b85384eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752439} /tidb/cdc/default/__cdc_meta__/capture/e5b2cf76-ec42-479f-8c58-0fff42a4aa91 {"id":"e5b2cf76-ec42-479f-8c58-0fff42a4aa91","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159b18 e5b2cf76-ec42-479f-8c58-0fff42a4aa91 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe159ba4 b2222a11-decb-457c-86eb-bc48b85384eb /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:26.406997872+08:00","start-ts":457114236043919370,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236057026569} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.514495707+08:00","start-ts":457114236083503115,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238049320972} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489359547990597012,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:40:26.613576086+08:00","start-ts":457114236109717518,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114236109717521} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114236581314565,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114236581314565,"min-table-barrier-ts":457114240041877512,"admin-job-type":0} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2222a11-decb-457c-86eb-bc48b85384eb/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b2cf76-ec42-479f-8c58-0fff42a4aa91/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359547990597012 {"id":7489359547990597012,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"address": "127.0.0.1:8300"' + set +x + tso='457114240110559233 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114240110559233 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:43 CST 2025] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.26822684.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114240144113666 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114240144113666 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:43 CST 2025] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.26872689.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "address": "127.0.0.1:8300", run task successfully pass test: list changefeed + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:44 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 {"id":"0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752441} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd1d8 0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 {"id":"0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752441} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd1d8 0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 {"id":"0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752441} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd1d8 0db3eb1d-02f0-44d1-b5ae-dfb506cda6d5 /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "readproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "insertproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "workload"="core" "mysql.port"="4000" "fieldcount"="3" "recordcount"="50" "updateproportion"="0" "scanproportion"="0" "mysql.db"="consistent_replicate_ddl" "readallfields"="true" "threadcount"="10" "operationcount"="0" ********************************************** Run finished, takes 14.413999ms INSERT - Takes(s): 0.0, Count: 50, OPS: 5989.0, Avg(us): 2706, Min(us): 1292, Max(us): 6529, 95th(us): 7000, 99th(us): 7000 pass test: get changefeed table consistent_replicate_ddl.usertable1 not exists for 1-th check, retry later pass test: pause changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:46 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/135e3c3d-b213-408a-8691-e5edab4710bf {"id":"135e3c3d-b213-408a-8691-e5edab4710bf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7d46ec 135e3c3d-b213-408a-8691-e5edab4710bf /tidb/cdc/default/default/upstream/7489359661149374804 {"id":7489359661149374804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/135e3c3d-b213-408a-8691-e5edab4710bf {"id":"135e3c3d-b213-408a-8691-e5edab4710bf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7d46ec 135e3c3d-b213-408a-8691-e5edab4710bf /tidb/cdc/default/default/upstream/7489359661149374804 {"id":7489359661149374804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/135e3c3d-b213-408a-8691-e5edab4710bf {"id":"135e3c3d-b213-408a-8691-e5edab4710bf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7d46ec 135e3c3d-b213-408a-8691-e5edab4710bf /tidb/cdc/default/default/upstream/7489359661149374804 {"id":7489359661149374804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2741.out cli changefeed create --start-ts=457114240110559233 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ pass test: resume changefeed pass test: rebalance table + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:46 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f19083dd-b4e8-4ca5-8603-eea376b57152 {"id":"f19083dd-b4e8-4ca5-8603-eea376b57152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7df9eb f19083dd-b4e8-4ca5-8603-eea376b57152 /tidb/cdc/default/default/upstream/7489359661716040254 {"id":7489359661716040254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f19083dd-b4e8-4ca5-8603-eea376b57152 {"id":"f19083dd-b4e8-4ca5-8603-eea376b57152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7df9eb f19083dd-b4e8-4ca5-8603-eea376b57152 /tidb/cdc/default/default/upstream/7489359661716040254 {"id":7489359661716040254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f19083dd-b4e8-4ca5-8603-eea376b57152 {"id":"f19083dd-b4e8-4ca5-8603-eea376b57152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752443} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7df9eb f19083dd-b4e8-4ca5-8603-eea376b57152 /tidb/cdc/default/default/upstream/7489359661716040254 {"id":7489359661716040254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2742.out cli changefeed create --start-ts=457114240144113666 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/foreign_key/conf/cf.toml Create changefeed successfully! ID: a9467c5a-d3c0-4ad4-9dd4-f532e6f9bd40 Info: {"upstream_id":7489359661149374804,"namespace":"default","id":"a9467c5a-d3c0-4ad4-9dd4-f532e6f9bd40","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:40:46.802406537+08:00","start_ts":457114240110559233,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114240110559233,"checkpoint_ts":457114240110559233,"checkpoint_time":"2025-04-04 15:40:41.828"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... pass test: move table Create changefeed successfully! ID: 33a2b88e-b466-4c72-91b1-a216d84b4a46 Info: {"upstream_id":7489359661716040254,"namespace":"default","id":"33a2b88e-b466-4c72-91b1-a216d84b4a46","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:40:46.902613159+08:00","start_ts":457114240144113666,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!foreign_key.t12"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114240144113666,"checkpoint_ts":457114240144113666,"checkpoint_time":"2025-04-04 15:40:41.956"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table consistent_replicate_ddl.usertable1 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.16 secs (494182840 bytes/sec) [Pipeline] { [Pipeline] cache + set +x + set +x table consistent_replicate_ddl.usertable1 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: get processors pass test: list processors pass test: set log level pass test: remove changefeed pass test: resign owner table consistent_replicate_ddl.usertable1 exists table consistent_replicate_ddl.usertable2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_ddl.usertable2 not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359639781440497 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb6efc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:3746, start at 2025-04-04 15:40:53.103912838 +0800 CST m=+24.136549767 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:53.111 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:53.105 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:53.105 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_ddl.usertable2 exists table consistent_replicate_ddl.usertable3 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359639781440497 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb6efc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:3746, start at 2025-04-04 15:40:53.103912838 +0800 CST m=+24.136549767 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:53.111 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:53.105 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:53.105 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359636725065301 TiDB Cluster ID. tikv_gc_leader_uuid 657fefa10f0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:3824, start at 2025-04-04 15:40:30.687528783 +0800 CST m=+1.635677980 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:30.694 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:30.652 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:30.652 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table consistent_replicate_ddl.usertable3 not exists for 2-th check, retry later [Fri Apr 4 15:40:57 CST 2025] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.51515153.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table foreign_key.finish_mark not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.70 secs (523567174 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 1-th check, retry later table consistent_replicate_ddl.usertable3 exists table consistent_replicate_ddl.usertable_bak not exists for 1-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later table foreign_key.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:00 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 {"id":"bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe6b9e3c bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 /tidb/cdc/default/default/upstream/7489359639781440497 {"id":7489359639781440497,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 {"id":"bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe6b9e3c bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 /tidb/cdc/default/default/upstream/7489359639781440497 {"id":7489359639781440497,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 {"id":"bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe6b9e3c bcf8b2dc-a1eb-45d3-8471-7930ecebcbf6 /tidb/cdc/default/default/upstream/7489359639781440497 {"id":7489359639781440497,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:41:00 CST 2025] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.52095211.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8400 failed: Connection refused * Failed to connect to 127.0.0.1 port 8400: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_ddl.usertable_bak not exists for 2-th check, retry later table row_format.finish_mark not exists for 3-th check, retry later table foreign_key.finish_mark not exists for 3-th check, retry later table consistent_replicate_ddl.usertable_bak exists table row_format.finish_mark not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8400 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41cd9a19-146e-4a1d-9cdf-7914dc2a813d {"id":"41cd9a19-146e-4a1d-9cdf-7914dc2a813d","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbe6bcc67 41cd9a19-146e-4a1d-9cdf-7914dc2a813d /tidb/cdc/default/default/upstream/7489359636725065301 {"id":7489359636725065301,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41cd9a19-146e-4a1d-9cdf-7914dc2a813d {"id":"41cd9a19-146e-4a1d-9cdf-7914dc2a813d","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbe6bcc67 41cd9a19-146e-4a1d-9cdf-7914dc2a813d /tidb/cdc/default/default/upstream/7489359636725065301 {"id":7489359636725065301,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41cd9a19-146e-4a1d-9cdf-7914dc2a813d {"id":"41cd9a19-146e-4a1d-9cdf-7914dc2a813d","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbe6bcc67 41cd9a19-146e-4a1d-9cdf-7914dc2a813d /tidb/cdc/default/default/upstream/7489359636725065301 {"id":7489359636725065301,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.5262.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306 -c test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/conf/up.toml table foreign_key.finish_mark not exists for 4-th check, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489359639781440497,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306","create_time":"2025-04-04T15:41:04.028633307+08:00","start_ts":457114245897912324,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114245897912324,"checkpoint_ts":457114245897912324,"checkpoint_time":"2025-04-04 15:41:03.905"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 5-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.5304.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:4000 -c test-2 --server http://127.0.0.1:8400 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/conf/down.toml Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489359636725065301,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:4000","create_time":"2025-04-04T15:41:05.875728728+08:00","start_ts":457114246395461635,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114246395461635,"checkpoint_ts":457114246395461635,"checkpoint_time":"2025-04-04 15:41:05.803"} PASS table foreign_key.finish_mark not exists for 5-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 6-th check, retry later + set +x table foreign_key.finish_mark not exists for 6-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.24 secs (489210776 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 7-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:09 CST 2025] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.28952897.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server2 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table foreign_key.finish_mark not exists for 7-th check, retry later table row_format.finish_mark not exists for 8-th check, retry later table foreign_key.finish_mark not exists for 8-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/b2266c37-2282-401c-aca6-4560b9fa80d0 {UpstreamID:7489359667513337815 Namespace:default ID:b2266c37-2282-401c-aca6-4560b9fa80d0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:44.29791299 +0800 CST StartTs:457114240733151237 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002f38240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114240746258439} {CheckpointTs:457114246356140038 MinTableBarrierTs:457114247941849089 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e930a9f9-90ef-4954-9d57-299a1a9ef83c {"id":"e930a9f9-90ef-4954-9d57-299a1a9ef83c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752469} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd31e e930a9f9-90ef-4954-9d57-299a1a9ef83c /tidb/cdc/default/default/changefeed/info/b2266c37-2282-401c-aca6-4560b9fa80d0 {"upstream-id":7489359667513337815,"namespace":"default","changefeed-id":"b2266c37-2282-401c-aca6-4560b9fa80d0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:44.29791299+08:00","start-ts":457114240733151237,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114240746258439} /tidb/cdc/default/default/changefeed/status/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":457114246356140038,"min-table-barrier-ts":457114247941849089,"admin-job-type":0} /tidb/cdc/default/default/task/position/e930a9f9-90ef-4954-9d57-299a1a9ef83c/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/b2266c37-2282-401c-aca6-4560b9fa80d0 {UpstreamID:7489359667513337815 Namespace:default ID:b2266c37-2282-401c-aca6-4560b9fa80d0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:44.29791299 +0800 CST StartTs:457114240733151237 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002f38240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114240746258439} {CheckpointTs:457114246356140038 MinTableBarrierTs:457114247941849089 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e930a9f9-90ef-4954-9d57-299a1a9ef83c {"id":"e930a9f9-90ef-4954-9d57-299a1a9ef83c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752469} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd31e e930a9f9-90ef-4954-9d57-299a1a9ef83c /tidb/cdc/default/default/changefeed/info/b2266c37-2282-401c-aca6-4560b9fa80d0 {"upstream-id":7489359667513337815,"namespace":"default","changefeed-id":"b2266c37-2282-401c-aca6-4560b9fa80d0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:44.29791299+08:00","start-ts":457114240733151237,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114240746258439} /tidb/cdc/default/default/changefeed/status/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":457114246356140038,"min-table-barrier-ts":457114247941849089,"admin-job-type":0} /tidb/cdc/default/default/task/position/e930a9f9-90ef-4954-9d57-299a1a9ef83c/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/b2266c37-2282-401c-aca6-4560b9fa80d0 {UpstreamID:7489359667513337815 Namespace:default ID:b2266c37-2282-401c-aca6-4560b9fa80d0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:44.29791299 +0800 CST StartTs:457114240733151237 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002f38240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114240746258439} {CheckpointTs:457114246356140038 MinTableBarrierTs:457114247941849089 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114246356140038, checkpointTs: 457114246356140038, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e930a9f9-90ef-4954-9d57-299a1a9ef83c {"id":"e930a9f9-90ef-4954-9d57-299a1a9ef83c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752469} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe7bd31e e930a9f9-90ef-4954-9d57-299a1a9ef83c /tidb/cdc/default/default/changefeed/info/b2266c37-2282-401c-aca6-4560b9fa80d0 {"upstream-id":7489359667513337815,"namespace":"default","changefeed-id":"b2266c37-2282-401c-aca6-4560b9fa80d0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:44.29791299+08:00","start-ts":457114240733151237,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114240746258439} /tidb/cdc/default/default/changefeed/status/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":457114246356140038,"min-table-barrier-ts":457114247941849089,"admin-job-type":0} /tidb/cdc/default/default/task/position/e930a9f9-90ef-4954-9d57-299a1a9ef83c/b2266c37-2282-401c-aca6-4560b9fa80d0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359667513337815 {"id":7489359667513337815,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table row_format.finish_mark not exists for 9-th check, retry later table foreign_key.finish_mark exists check diff successfully table foreign_key.t12 does not exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table row_format.finish_mark not exists for 10-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:15 CST 2025] <<<<<< run test case foreign_key success! >>>>>> table row_format.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:18 CST 2025] <<<<<< run test case row_format success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.67 secs (464855563 bytes/sec) [Pipeline] { [Pipeline] cache pass test: get tso wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:41:22 CST 2025] <<<<<< run test case http_api success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.54 secs (534206065 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G06 [Pipeline] sh Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-xh4jw HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=0c61ca9b-3f24-4af0-afc8-6e2451e01e5f GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G06 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz pingcap_tiflow_pull_cdc_integration_test_3828-xh4jw PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G05 [Pipeline] sh Run cases: charset_gbk ddl_manager multi_source vector GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-l1r4j HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=c013ee30-0e32-4693-a26c-bde356c2ec08 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G05 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-l1r4j pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G08 [Pipeline] sh Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-t3v13 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=19d3d64b-b0e3-4d90-ae6e-65c9a5b3eac5 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G08 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-t3v13 pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G07 [Pipeline] sh Run cases: kv_client_stream_reconnect cdc split_region GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-23b4k HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=7c0e05d8-7c3c-400c-b351-88ef815ab1cd GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G07 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-23b4k pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G10 Run cases: default_value simple cdc_server_tips event_filter sql_mode GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-zl944 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=7f8e8f49-6963-4109-ac13-6d8478af91c4 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G10 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-zl944 pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: mysql... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G09 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-7dz70 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=ef3d5bb9-07d9-40e0-96b3-836cc45ea29d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G09 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-7dz70 pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... [Pipeline] { Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] stage Verifying downstream PD is started... [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Verifying downstream PD is started... [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] cache Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359922160383962 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe1ba40001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:3897, start at 2025-04-04 15:41:36.890246183 +0800 CST m=+2.286578324 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:36.957 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:36.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:36.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359922160383962 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe1ba40001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:3897, start at 2025-04-04 15:41:36.890246183 +0800 CST m=+2.286578324 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:36.957 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:36.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:36.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359919394908244 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe2510000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:3956, start at 2025-04-04 15:41:37.50984532 +0800 CST m=+2.838176934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:37.532 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:37.526 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:37.526 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5169.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359942132040656 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe59840007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:1333, start at 2025-04-04 15:41:40.844631151 +0800 CST m=+2.591932256 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.853 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.833 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.833 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359942132040656 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe59840007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:1333, start at 2025-04-04 15:41:40.844631151 +0800 CST m=+2.591932256 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.853 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.833 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.833 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359942296493193 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe5b080002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:1409, start at 2025-04-04 15:41:40.931502312 +0800 CST m=+2.581584828 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.941 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.930 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.930 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359934960688933 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe53100005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:1350, start at 2025-04-04 15:41:40.428014612 +0800 CST m=+2.411025589 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.468 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.470 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359934960688933 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe53100005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:1350, start at 2025-04-04 15:41:40.428014612 +0800 CST m=+2.411025589 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.468 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.470 +0800 The time when last GC starts. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT tikv_gc_safe_point 20250404-15:31:40.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) bootstrapped True Bootstrap flag. Do not delete. Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT tidb_server_version 246 Bootstrap version. Do not delete. bootstrapped True Bootstrap flag. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359936205000404 TiDB Cluster ID. cluster_id 7489359937472388290 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe54ec0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_uuid 657fefe56200009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:3900, start at 2025-04-04 15:41:40.555697315 +0800 CST m=+2.438392048 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:1416, start at 2025-04-04 15:41:40.661962701 +0800 CST m=+2.571673394 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.565 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.670 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.539 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_last_run_time 20250404-15:41:40.667 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.539 +0800 All versions after safe point can be accessed. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.667 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash VARIABLE_NAME VARIABLE_VALUE COMMENT Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 bootstrapped True Bootstrap flag. Do not delete. Edition: Community tidb_server_version 246 Bootstrap version. Do not delete. Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD system_tz Asia/Shanghai TiDB Global System Timezone. UTC Build Time: 2025-04-03 03:49:11 new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359936205000404 TiDB Cluster ID. Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake tikv_gc_leader_uuid 657fefe54ec0009 Current GC worker leader UUID. (DO NOT EDIT) Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:3900, start at 2025-04-04 15:41:40.555697315 +0800 CST m=+2.438392048 Host name and pid of current GC leader. (DO NOT EDIT) Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... tikv_gc_leader_lease 20250404-15:43:40.565 +0800 Current GC worker leader lease. (DO NOT EDIT) Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.539 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:40.539 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359937241893480 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe5674000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:3976, start at 2025-04-04 15:41:40.67804698 +0800 CST m=+2.488950445 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:40.687 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:40.687 +0800 The time when last GC starts. (DO NOT EDIT) arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } tikv_gc_safe_point 20250404-15:31:40.687 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114255719923713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114255719923713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359934995950678 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe5e04000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:1282, start at 2025-04-04 15:41:41.139086729 +0800 CST m=+2.246104850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:41.151 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:41.171 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:41.171 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359934995950678 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe5e04000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:1282, start at 2025-04-04 15:41:41.139086729 +0800 CST m=+2.246104850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:41.151 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:41.171 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:41.171 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359942241992100 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe5ee80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:1361, start at 2025-04-04 15:41:41.209428539 +0800 CST m=+2.239901265 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:41.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:41.178 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:41.178 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:41:43 CST 2025] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.52175219.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359946153784816 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe6d38002b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:1328, start at 2025-04-04 15:41:42.104057097 +0800 CST m=+2.297735057 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.114 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.153 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.153 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359946153784816 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe6d38002b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:1328, start at 2025-04-04 15:41:42.104057097 +0800 CST m=+2.297735057 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.114 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.153 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.153 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359935030977840 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe6b90000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:1402, start at 2025-04-04 15:41:42.013679694 +0800 CST m=+2.122062467 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.060 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.053 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.053 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:41:44 CST 2025] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26752677.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5208.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2633.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359941920197302 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe80fc0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:1275, start at 2025-04-04 15:41:43.364162982 +0800 CST m=+2.638614334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:43.371 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:43.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:43.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359941920197302 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe80fc0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:1275, start at 2025-04-04 15:41:43.364162982 +0800 CST m=+2.638614334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:43.371 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:43.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:43.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359950475536735 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe8220000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:1357, start at 2025-04-04 15:41:43.453661595 +0800 CST m=+2.611867132 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:43.461 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:43.456 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:43.456 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2675.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359945637189059 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe985c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:1345, start at 2025-04-04 15:41:44.903226568 +0800 CST m=+3.692585313 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:44.915 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:44.905 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:44.905 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359945637189059 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe985c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:1345, start at 2025-04-04 15:41:44.903226568 +0800 CST m=+3.692585313 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:44.915 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:44.905 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:44.905 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359951459511680 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe99340005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:1417, start at 2025-04-04 15:41:44.917532118 +0800 CST m=+3.625041966 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:44.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:45.010 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:45.010 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table tidb_cdc.syncpoint_v1 exists table tidb_cdc.syncpoint_v1 exists table bdr_mode.finish_mark exists table bdr_mode.finish_mark not exists for 1-th check, retry later table bdr_mode.finish_mark not exists for 2-th check, retry later table bdr_mode.finish_mark exists check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... + set +x + tso='457114256654991361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114256654991361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:46 CST 2025] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.52455247.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114256679370753 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114256679370753 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:46 CST 2025] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.26732675.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 6.78 secs (594575836 bytes/sec) [Pipeline] { [Pipeline] cache + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... [Fri Apr 4 15:41:47 CST 2025] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.26472649.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114256833511425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114256833511425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "requestdistribution"="uniform" "readproportion"="0" "mysql.port"="4000" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "readallfields"="true" "operationcount"="0" "recordcount"="10" "threadcount"="2" "dotransactions"="false" "insertproportion"="0" "workload"="core" "mysql.db"="sink_retry" "scanproportion"="0" ********************************************** Run finished, takes 9.891539ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1916.6, Avg(us): 1892, Min(us): 1054, Max(us): 4620, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:41:47 CST 2025] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.27322734.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:47 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12792f43-55f4-4f7a-a865-60b914a000da {"id":"12792f43-55f4-4f7a-a865-60b914a000da","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf661400 12792f43-55f4-4f7a-a865-60b914a000da /tidb/cdc/default/default/upstream/7489359922160383962 {"id":7489359922160383962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12792f43-55f4-4f7a-a865-60b914a000da {"id":"12792f43-55f4-4f7a-a865-60b914a000da","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf661400 12792f43-55f4-4f7a-a865-60b914a000da /tidb/cdc/default/default/upstream/7489359922160383962 {"id":7489359922160383962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12792f43-55f4-4f7a-a865-60b914a000da {"id":"12792f43-55f4-4f7a-a865-60b914a000da","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf661400 12792f43-55f4-4f7a-a865-60b914a000da /tidb/cdc/default/default/upstream/7489359922160383962 {"id":7489359922160383962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5274.out cli changefeed create --start-ts=457114255719923713 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:47 CST 2025] <<<<<< run test case bdr_mode success! >>>>>> Create changefeed successfully! ID: 3172ec97-36b5-4226-add1-814b94264842 Info: {"upstream_id":7489359922160383962,"namespace":"default","id":"3172ec97-36b5-4226-add1-814b94264842","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:47.45390709+08:00","start_ts":457114255719923713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114255719923713,"checkpoint_ts":457114255719923713,"checkpoint_time":"2025-04-04 15:41:41.373"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:47 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47da71e9-97af-4ad3-b86f-b041deb459b6 {"id":"47da71e9-97af-4ad3-b86f-b041deb459b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7b28d7 47da71e9-97af-4ad3-b86f-b041deb459b6 /tidb/cdc/default/default/upstream/7489359934960688933 {"id":7489359934960688933,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47da71e9-97af-4ad3-b86f-b041deb459b6 {"id":"47da71e9-97af-4ad3-b86f-b041deb459b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7b28d7 47da71e9-97af-4ad3-b86f-b041deb459b6 /tidb/cdc/default/default/upstream/7489359934960688933 {"id":7489359934960688933,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47da71e9-97af-4ad3-b86f-b041deb459b6 {"id":"47da71e9-97af-4ad3-b86f-b041deb459b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7b28d7 47da71e9-97af-4ad3-b86f-b041deb459b6 /tidb/cdc/default/default/upstream/7489359934960688933 {"id":7489359934960688933,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 4dfb0608-f527-4e06-ae81-c69171e7ca09 Info: {"upstream_id":7489359934960688933,"namespace":"default","id":"4dfb0608-f527-4e06-ae81-c69171e7ca09","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:47.76389728+08:00","start_ts":457114256544890881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114256544890881,"checkpoint_ts":457114256544890881,"checkpoint_time":"2025-04-04 15:41:44.520"} [Fri Apr 4 15:41:47 CST 2025] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.26562658.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359965509074212 TiDB Cluster ID. tikv_gc_leader_uuid 657fefebe480007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:4181, start at 2025-04-04 15:41:47.293112953 +0800 CST m=+3.046771487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:47.333 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:47.332 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:47.332 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359965509074212 TiDB Cluster ID. tikv_gc_leader_uuid 657fefebe480007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:4181, start at 2025-04-04 15:41:47.293112953 +0800 CST m=+3.046771487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:47.333 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:47.332 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:47.332 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359956924252682 TiDB Cluster ID. tikv_gc_leader_uuid 657fefec1540009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:4244, start at 2025-04-04 15:41:47.490552711 +0800 CST m=+3.141613034 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:47.536 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:47.627 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:47.627 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/129c7335-4537-42ec-b604-8a534eae59a3 {"id":"129c7335-4537-42ec-b604-8a534eae59a3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf75c2e5 129c7335-4537-42ec-b604-8a534eae59a3 /tidb/cdc/default/default/upstream/7489359936205000404 {"id":7489359936205000404,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/129c7335-4537-42ec-b604-8a534eae59a3 {"id":"129c7335-4537-42ec-b604-8a534eae59a3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf75c2e5 129c7335-4537-42ec-b604-8a534eae59a3 /tidb/cdc/default/default/upstream/7489359936205000404 {"id":7489359936205000404,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/129c7335-4537-42ec-b604-8a534eae59a3 {"id":"129c7335-4537-42ec-b604-8a534eae59a3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf75c2e5 129c7335-4537-42ec-b604-8a534eae59a3 /tidb/cdc/default/default/upstream/7489359936205000404 {"id":7489359936205000404,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5298.out cli changefeed create --start-ts=457114256654991361 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: e84fc415-a4c0-4efa-bbad-454db146906b Info: {"upstream_id":7489359936205000404,"namespace":"default","id":"e84fc415-a4c0-4efa-bbad-454db146906b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:49.910508123+08:00","start_ts":457114256654991361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114256654991361,"checkpoint_ts":457114256654991361,"checkpoint_time":"2025-04-04 15:41:44.940"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e8afa4-5d39-4739-9382-7d9c3aa5ca18 {"id":"78e8afa4-5d39-4739-9382-7d9c3aa5ca18","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7e23f6 78e8afa4-5d39-4739-9382-7d9c3aa5ca18 /tidb/cdc/default/default/upstream/7489359942132040656 {"id":7489359942132040656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e8afa4-5d39-4739-9382-7d9c3aa5ca18 {"id":"78e8afa4-5d39-4739-9382-7d9c3aa5ca18","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7e23f6 78e8afa4-5d39-4739-9382-7d9c3aa5ca18 /tidb/cdc/default/default/upstream/7489359942132040656 {"id":7489359942132040656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e8afa4-5d39-4739-9382-7d9c3aa5ca18 {"id":"78e8afa4-5d39-4739-9382-7d9c3aa5ca18","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752506} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7e23f6 78e8afa4-5d39-4739-9382-7d9c3aa5ca18 /tidb/cdc/default/default/upstream/7489359942132040656 {"id":7489359942132040656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2730.out cli changefeed create --start-ts=457114256679370753 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: a3d947eb-94ed-437b-9034-1ad9bca7525d Info: {"upstream_id":7489359942132040656,"namespace":"default","id":"a3d947eb-94ed-437b-9034-1ad9bca7525d","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:50.146721954+08:00","start_ts":457114256679370753,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114256679370753,"checkpoint_ts":457114256679370753,"checkpoint_time":"2025-04-04 15:41:45.033"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b009a972-aa22-43eb-869b-327032c98d9b {"id":"b009a972-aa22-43eb-869b-327032c98d9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7fcbda b009a972-aa22-43eb-869b-327032c98d9b /tidb/cdc/default/default/upstream/7489359941920197302 {"id":7489359941920197302,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b009a972-aa22-43eb-869b-327032c98d9b {"id":"b009a972-aa22-43eb-869b-327032c98d9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7fcbda b009a972-aa22-43eb-869b-327032c98d9b /tidb/cdc/default/default/upstream/7489359941920197302 {"id":7489359941920197302,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b009a972-aa22-43eb-869b-327032c98d9b {"id":"b009a972-aa22-43eb-869b-327032c98d9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7fcbda b009a972-aa22-43eb-869b-327032c98d9b /tidb/cdc/default/default/upstream/7489359941920197302 {"id":7489359941920197302,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41d07c4f-e677-4a42-b69c-56fe3df1cf49 {"id":"41d07c4f-e677-4a42-b69c-56fe3df1cf49","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a91f8 41d07c4f-e677-4a42-b69c-56fe3df1cf49 /tidb/cdc/default/default/upstream/7489359934995950678 {"id":7489359934995950678,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41d07c4f-e677-4a42-b69c-56fe3df1cf49 {"id":"41d07c4f-e677-4a42-b69c-56fe3df1cf49","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a91f8 41d07c4f-e677-4a42-b69c-56fe3df1cf49 /tidb/cdc/default/default/upstream/7489359934995950678 {"id":7489359934995950678,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41d07c4f-e677-4a42-b69c-56fe3df1cf49 {"id":"41d07c4f-e677-4a42-b69c-56fe3df1cf49","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a91f8 41d07c4f-e677-4a42-b69c-56fe3df1cf49 /tidb/cdc/default/default/upstream/7489359934995950678 {"id":7489359934995950678,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2785.out cli changefeed create --start-ts=457114256833511425 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' 0 check diff failed 1-th time, retry later Create changefeed successfully! ID: 5fe46d25-379e-4e57-a42a-fbad1620537b Info: {"upstream_id":7489359934995950678,"namespace":"default","id":"5fe46d25-379e-4e57-a42a-fbad1620537b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:50.717543949+08:00","start_ts":457114256833511425,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114256833511425,"checkpoint_ts":457114256833511425,"checkpoint_time":"2025-04-04 15:41:45.621"} PASS The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting TLS PD... coverage: 2.4% of statements in github.com/pingcap/tiflow/... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40f96d62-36c9-4873-9f06-65d997651902 {"id":"40f96d62-36c9-4873-9f06-65d997651902","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752508} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8156d8 40f96d62-36c9-4873-9f06-65d997651902 /tidb/cdc/default/default/upstream/7489359945637189059 {"id":7489359945637189059,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40f96d62-36c9-4873-9f06-65d997651902 {"id":"40f96d62-36c9-4873-9f06-65d997651902","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752508} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8156d8 40f96d62-36c9-4873-9f06-65d997651902 /tidb/cdc/default/default/upstream/7489359945637189059 {"id":7489359945637189059,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40f96d62-36c9-4873-9f06-65d997651902 {"id":"40f96d62-36c9-4873-9f06-65d997651902","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752508} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8156d8 40f96d62-36c9-4873-9f06-65d997651902 /tidb/cdc/default/default/upstream/7489359945637189059 {"id":7489359945637189059,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table drop_tables.c not exists for 1-th check, retry later + set +x succeed to verify meta placement rules TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later + set +x check diff failed 2-th time, retry later table drop_tables.c not exists for 2-th check, retry later [Fri Apr 4 15:41:52 CST 2025] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27242726.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later check diff successfully check_safepoint_forward http://127.0.0.1:2379 7489359941920197302 457114258966839302 457114258075287555 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:55 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/10c1fa01-7af3-4cd1-b00b-f1989983c3e8 {"id":"10c1fa01-7af3-4cd1-b00b-f1989983c3e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752512} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a7962 10c1fa01-7af3-4cd1-b00b-f1989983c3e8 /tidb/cdc/default/default/upstream/7489359946153784816 {"id":7489359946153784816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/10c1fa01-7af3-4cd1-b00b-f1989983c3e8 {"id":"10c1fa01-7af3-4cd1-b00b-f1989983c3e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752512} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a7962 10c1fa01-7af3-4cd1-b00b-f1989983c3e8 /tidb/cdc/default/default/upstream/7489359946153784816 {"id":7489359946153784816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/10c1fa01-7af3-4cd1-b00b-f1989983c3e8 {"id":"10c1fa01-7af3-4cd1-b00b-f1989983c3e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752512} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf7a7962 10c1fa01-7af3-4cd1-b00b-f1989983c3e8 /tidb/cdc/default/default/upstream/7489359946153784816 {"id":7489359946153784816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.11 secs (566690481 bytes/sec) [Pipeline] { [Pipeline] cache check_changefeed_state http://127.0.0.1:2379 973d7db1-abc6-4316-af83-da2ade747b97 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=973d7db1-abc6-4316-af83-da2ade747b97 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 973d7db1-abc6-4316-af83-da2ade747b97 -s run task successfully table drop_tables.c not exists for 3-th check, retry later + info='{ "upstream_id": 7489359946153784816, "namespace": "default", "id": "973d7db1-abc6-4316-af83-da2ade747b97", "state": "normal", "checkpoint_tso": 457114259447873542, "checkpoint_time": "2025-04-04 15:41:55.594", "error": null }' + echo '{ "upstream_id": 7489359946153784816, "namespace": "default", "id": "973d7db1-abc6-4316-af83-da2ade747b97", "state": "normal", "checkpoint_tso": 457114259447873542, "checkpoint_time": "2025-04-04 15:41:55.594", "error": null }' { "upstream_id": 7489359946153784816, "namespace": "default", "id": "973d7db1-abc6-4316-af83-da2ade747b97", "state": "normal", "checkpoint_tso": 457114259447873542, "checkpoint_time": "2025-04-04 15:41:55.594", "error": null } ++ echo '{' '"upstream_id":' 7489359946153784816, '"namespace":' '"default",' '"id":' '"973d7db1-abc6-4316-af83-da2ade747b97",' '"state":' '"normal",' '"checkpoint_tso":' 457114259447873542, '"checkpoint_time":' '"2025-04-04' '15:41:55.594",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359946153784816, '"namespace":' '"default",' '"id":' '"973d7db1-abc6-4316-af83-da2ade747b97",' '"state":' '"normal",' '"checkpoint_tso":' 457114259447873542, '"checkpoint_time":' '"2025-04-04' '15:41:55.594",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_changefeed_state http://127.0.0.1:2379 af6c709d-f10f-4d65-bb43-947be916e138 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=af6c709d-f10f-4d65-bb43-947be916e138 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c af6c709d-f10f-4d65-bb43-947be916e138 -s check data successfully check diff failed 1-th time, retry later + info='{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' + echo '{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' { "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null } ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"stopped",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"stopped",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359941920197302 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:57 CST 2025] <<<<<< run test case simple success! >>>>>> table drop_tables.c not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360014899894459 TiDB Cluster ID. tikv_gc_leader_uuid 657feff5ff0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:5892, start at 2025-04-04 15:41:57.665401951 +0800 CST m=+1.676474661 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:57.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:57.678 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:57.678 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Fri Apr 4 15:41:58 CST 2025] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.59385940.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later run task successfully check_changefeed_state http://127.0.0.1:2379 af6c709d-f10f-4d65-bb43-947be916e138 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=af6c709d-f10f-4d65-bb43-947be916e138 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c af6c709d-f10f-4d65-bb43-947be916e138 -s table drop_tables.c not exists for 5-th check, retry later + info='{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "normal", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' + echo '{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "normal", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' { "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "normal", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null } ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"normal",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"normal",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7489359941920197302 457114259491127301 457114259491127302 start tidb cluster in /tmp/tidb_cdc_test/capture_suicide_while_balance_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff successfully run task successfully check_changefeed_state http://127.0.0.1:2379 af6c709d-f10f-4d65-bb43-947be916e138 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=af6c709d-f10f-4d65-bb43-947be916e138 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c af6c709d-f10f-4d65-bb43-947be916e138 -s wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:01 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/default/upstream/7489360014899894459 {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/default/upstream/7489360014899894459 {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/default/upstream/7489360014899894459 {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x + info='{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' + echo '{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null }' { "upstream_id": 7489359941920197302, "namespace": "default", "id": "af6c709d-f10f-4d65-bb43-947be916e138", "state": "stopped", "checkpoint_tso": 457114259491127302, "checkpoint_time": "2025-04-04 15:41:55.759", "error": null } ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"stopped",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"af6c709d-f10f-4d65-bb43-947be916e138",' '"state":' '"stopped",' '"checkpoint_tso":' 457114259491127302, '"checkpoint_time":' '"2025-04-04' '15:41:55.759",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 efbb9966-7928-4385-ad39-e7f5afc9325f normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=efbb9966-7928-4385-ad39-e7f5afc9325f + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c efbb9966-7928-4385-ad39-e7f5afc9325f -s table drop_tables.c exists + info='{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "efbb9966-7928-4385-ad39-e7f5afc9325f", "state": "normal", "checkpoint_tso": 457114260998193153, "checkpoint_time": "2025-04-04 15:42:01.508", "error": null }' + echo '{ "upstream_id": 7489359941920197302, "namespace": "default", "id": "efbb9966-7928-4385-ad39-e7f5afc9325f", "state": "normal", "checkpoint_tso": 457114260998193153, "checkpoint_time": "2025-04-04 15:42:01.508", "error": null }' { "upstream_id": 7489359941920197302, "namespace": "default", "id": "efbb9966-7928-4385-ad39-e7f5afc9325f", "state": "normal", "checkpoint_tso": 457114260998193153, "checkpoint_time": "2025-04-04 15:42:01.508", "error": null } ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"efbb9966-7928-4385-ad39-e7f5afc9325f",' '"state":' '"normal",' '"checkpoint_tso":' 457114260998193153, '"checkpoint_time":' '"2025-04-04' '15:42:01.508",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359941920197302, '"namespace":' '"default",' '"id":' '"efbb9966-7928-4385-ad39-e7f5afc9325f",' '"state":' '"normal",' '"checkpoint_tso":' 457114260998193153, '"checkpoint_time":' '"2025-04-04' '15:42:01.508",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359941920197302 check diff successfully wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:02 CST 2025] <<<<<< run test case processor_err_chan success! >>>>>> table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 2-th check, retry later [Fri Apr 4 15:42:03 CST 2025] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.60116013.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:03 CST 2025] <<<<<< run test case drop_many_tables success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.finish_mark exists check table exists success Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.42 secs (543177743 bytes/sec) [Pipeline] { run task successfully [Pipeline] cache check diff successfully Changefeed remove successfully. ID: af6c709d-f10f-4d65-bb43-947be916e138 CheckpointTs: 457114259491127302 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_forward http://127.0.0.1:2379 7489359941920197302 457114259491127301 457114259491127302 457114260998193153 wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... run task successfully + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:06 GMT < Content-Length: 1661 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/capture/2909d3de-fed2-4449-b8a4-075bfc5f5f4a {"id":"2909d3de-fed2-4449-b8a4-075bfc5f5f4a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd9018 2909d3de-fed2-4449-b8a4-075bfc5f5f4a /tidb/cdc/default/default/upstream/7489360014899894459 {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/capture/2909d3de-fed2-4449-b8a4-075bfc5f5f4a {"id":"2909d3de-fed2-4449-b8a4-075bfc5f5f4a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd9018 2909d3de-fed2-4449-b8a4-075bfc5f5f4a /tidb/cdc/default/default/upstream/7489360014899894459 + grep -q 'failed to get info:' {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a475195-13bc-4985-8d33-dc91ce8c0223 {"id":"0a475195-13bc-4985-8d33-dc91ce8c0223","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/capture/2909d3de-fed2-4449-b8a4-075bfc5f5f4a {"id":"2909d3de-fed2-4449-b8a4-075bfc5f5f4a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd8fdb 0a475195-13bc-4985-8d33-dc91ce8c0223 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbfbd9018 2909d3de-fed2-4449-b8a4-075bfc5f5f4a /tidb/cdc/default/default/upstream/7489360014899894459 {"id":7489360014899894459,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:06 CST 2025] <<<<<< run test case charset_gbk success! >>>>>> Changefeed remove successfully. ID: efbb9966-7928-4385-ad39-e7f5afc9325f CheckpointTs: 457114262112305158 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_cleared http://127.0.0.1:2379 7489359941920197302 run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: check health pass test: get status cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:08 CST 2025] <<<<<< run test case gc_safepoint success! >>>>>> pass test: create changefeed check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s + info='{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114262890610696, "checkpoint_time": "2025-04-04 15:42:08.727", "error": null }' + echo '{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114262890610696, "checkpoint_time": "2025-04-04 15:42:08.727", "error": null }' { "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114262890610696, "checkpoint_time": "2025-04-04 15:42:08.727", "error": null } ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114262890610696, '"checkpoint_time":' '"2025-04-04' '15:42:08.727",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114262890610696, '"checkpoint_time":' '"2025-04-04' '15:42:08.727",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s + info='{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114262943301639, "checkpoint_time": "2025-04-04 15:42:08.928", "error": null }' + echo '{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114262943301639, "checkpoint_time": "2025-04-04 15:42:08.928", "error": null }' { "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114262943301639, "checkpoint_time": "2025-04-04 15:42:08.928", "error": null } ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114262943301639, '"checkpoint_time":' '"2025-04-04' '15:42:08.928",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114262943301639, '"checkpoint_time":' '"2025-04-04' '15:42:08.928",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s + info='{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null }' + echo '{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null }' { "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null } ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114263244505090, '"checkpoint_time":' '"2025-04-04' '15:42:10.077",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114263244505090, '"checkpoint_time":' '"2025-04-04' '15:42:10.077",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s + info='{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null }' + echo '{ "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null }' { "upstream_id": 7489360014899894459, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114263244505090, "checkpoint_time": "2025-04-04 15:42:10.077", "error": null } ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114263244505090, '"checkpoint_time":' '"2025-04-04' '15:42:10.077",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360014899894459, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114263244505090, '"checkpoint_time":' '"2025-04-04' '15:42:10.077",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360062626835898 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0009680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:6350, start at 2025-04-04 15:42:08.478922912 +0800 CST m=+2.341722653 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:08.487 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:08.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:08.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360062626835898 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0009680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:6350, start at 2025-04-04 15:42:08.478922912 +0800 CST m=+2.341722653 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:08.487 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:08.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:08.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360061956053330 TiDB Cluster ID. tikv_gc_leader_uuid 657ff000964000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:6431, start at 2025-04-04 15:42:08.49006914 +0800 CST m=+2.243321118 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:08.531 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:08.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:08.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Fri Apr 4 15:42:12 CST 2025] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.77777779.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.verify_table_eligible not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.73 secs (521446032 bytes/sec) [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.verify_table_eligible not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:15 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:42:15 CST 2025] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.78277829.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.verify_table_eligible exists pass test: list changefeed =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: get changefeed + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:18 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e4ff75f-ce79-47da-85b6-576597a2e0eb {"id":"5e4ff75f-ce79-47da-85b6-576597a2e0eb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe34612 5e4ff75f-ce79-47da-85b6-576597a2e0eb /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e4ff75f-ce79-47da-85b6-576597a2e0eb {"id":"5e4ff75f-ce79-47da-85b6-576597a2e0eb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe34612 5e4ff75f-ce79-47da-85b6-576597a2e0eb /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e4ff75f-ce79-47da-85b6-576597a2e0eb {"id":"5e4ff75f-ce79-47da-85b6-576597a2e0eb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/capture/a9a3317d-06a5-4968-bac2-d133ef482a50 {"id":"a9a3317d-06a5-4968-bac2-d133ef482a50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752532} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe345d6 a9a3317d-06a5-4968-bac2-d133ef482a50 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe34612 5e4ff75f-ce79-47da-85b6-576597a2e0eb /tidb/cdc/default/default/upstream/7489360062626835898 {"id":7489360062626835898,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_suicide_while_balance_table.t1 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360104063771023 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00b6140006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:3761, start at 2025-04-04 15:42:19.527682713 +0800 CST m=+2.872033044 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:19.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:19.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:19.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360104063771023 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00b6140006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:3761, start at 2025-04-04 15:42:19.527682713 +0800 CST m=+2.872033044 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:19.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:19.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:19.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360104015658294 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00b6c80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:3826, start at 2025-04-04 15:42:19.581396688 +0800 CST m=+2.830964428 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:19.632 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:19.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:19.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table capture_suicide_while_balance_table.t1 exists table capture_suicide_while_balance_table.t2 not exists for 1-th check, retry later pass test: pause changefeed start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: update changefeed VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360112277712112 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00d2a40035 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:3918, start at 2025-04-04 15:42:21.371126348 +0800 CST m=+2.416722127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:21.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:21.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:21.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360112277712112 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00d2a40035 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:3918, start at 2025-04-04 15:42:21.371126348 +0800 CST m=+2.416722127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:21.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:21.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:21.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360115865257827 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00d2cc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:3967, start at 2025-04-04 15:42:21.409293517 +0800 CST m=+2.388681627 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:21.459 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:21.463 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:21.463 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5167.out cli tso query --pd=http://127.0.0.1:2379 table capture_suicide_while_balance_table.t2 exists table capture_suicide_while_balance_table.t3 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.21 secs (558644760 bytes/sec) [Pipeline] { [Pipeline] cache table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "dotransactions"="false" "mysql.user"="root" "mysql.port"="4000" "updateproportion"="0" "operationcount"="0" "threadcount"="2" "scanproportion"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="sink_retry" "requestdistribution"="uniform" "workload"="core" "recordcount"="10" "readallfields"="true" "insertproportion"="0" ********************************************** Run finished, takes 6.618471ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2457.1, Avg(us): 1204, Min(us): 613, Max(us): 2963, 95th(us): 3000, 99th(us): 3000 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:24 CST 2025] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: resume changefeed + set +x + tso='457114266796032001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114266796032001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_suicide_while_balance_table.t3 exists table capture_suicide_while_balance_table.t4 not exists for 1-th check, retry later try a VALID cdc server command [Fri Apr 4 15:42:25 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.52105212.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:42:25 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.52625264.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 pass test: rebalance table Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360124401974112 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00fe98000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:6414, start at 2025-04-04 15:42:24.197592906 +0800 CST m=+2.205488498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.240 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360124401974112 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00fe98000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:6414, start at 2025-04-04 15:42:24.197592906 +0800 CST m=+2.205488498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.240 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360121656358726 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00fe8c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:6487, start at 2025-04-04 15:42:24.180072674 +0800 CST m=+2.112981475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.187 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.163 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.163 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } pass test: list processors ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_suicide_while_balance_table.t4 exists Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:28 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aba8076-26e7-4846-86e3-6a61323b5b82 {"id":"4aba8076-26e7-4846-86e3-6a61323b5b82","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b20d9 4aba8076-26e7-4846-86e3-6a61323b5b82 /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aba8076-26e7-4846-86e3-6a61323b5b82 {"id":"4aba8076-26e7-4846-86e3-6a61323b5b82","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b20d9 4aba8076-26e7-4846-86e3-6a61323b5b82 /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aba8076-26e7-4846-86e3-6a61323b5b82 {"id":"4aba8076-26e7-4846-86e3-6a61323b5b82","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b20d9 4aba8076-26e7-4846-86e3-6a61323b5b82 /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "operationcount"="0" "insertproportion"="0" "mysql.port"="4000" "workload"="core" "requestdistribution"="uniform" "dotransactions"="false" "mysql.db"="changefeed_reconstruct" "readproportion"="0" "scanproportion"="0" "mysql.user"="root" "threadcount"="4" "recordcount"="50" "updateproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 16.633831ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3836.8, Avg(us): 1335, Min(us): 927, Max(us): 4226, 95th(us): 5000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:28 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 {"id":"d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc00a56f4 d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 /tidb/cdc/default/default/upstream/7489360104063771023 {"id":7489360104063771023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 {"id":"d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 [Fri Apr 4 15:42:28 CST 2025] <<<<<< START cdc server in processor_stop_delay case >>>>>> /tidb/cdc/default/__cdc_meta__/owner/223195ffc00a56f4 d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 + [[ '' == \t\r\u\e ]] + set +e /tidb/cdc/default/default/upstream/7489360104063771023 + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' {"id":7489360104063771023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + echo ' *** owner info ***: *** processors info ***: + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.77087710.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) {"id":"d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752545} ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 * Failed to connect to 127.0.0.1 port 8300: Connection refused /tidb/cdc/default/__cdc_meta__/owner/223195ffc00a56f4 * Closing connection 0 + res= + echo '' d6f32b08-0f9e-43fc-a26f-d67d2b6e80b6 /tidb/cdc/default/default/upstream/7489360104063771023 + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 {"id":7489360104063771023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360144171780271 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0130a00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:3864, start at 2025-04-04 15:42:27.404342153 +0800 CST m=+2.209586693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:27.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:27.459 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:27.459 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360144171780271 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0130a00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:3864, start at 2025-04-04 15:42:27.404342153 +0800 CST m=+2.209586693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:27.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:27.459 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:27.459 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360140784537588 TiDB Cluster ID. tikv_gc_leader_uuid 657ff013238000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:3918, start at 2025-04-04 15:42:27.497468373 +0800 CST m=+2.224332411 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:27.506 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:27.470 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:27.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 140 100 35 100 105 686 2058 --:--:-- --:--:-- --:--:-- 2745 { "status": true, "message": "" table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later pass test: get processor + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:31 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1c82f79-2c53-4069-972f-7a362554ba3f {"id":"a1c82f79-2c53-4069-972f-7a362554ba3f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752548} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0231cd5 a1c82f79-2c53-4069-972f-7a362554ba3f /tidb/cdc/default/default/upstream/7489360124401974112 {"id":7489360124401974112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1c82f79-2c53-4069-972f-7a362554ba3f {"id":"a1c82f79-2c53-4069-972f-7a362554ba3f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752548} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0231cd5 a1c82f79-2c53-4069-972f-7a362554ba3f /tidb/cdc/default/default/upstream/7489360124401974112 {"id":7489360124401974112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1c82f79-2c53-4069-972f-7a362554ba3f {"id":"a1c82f79-2c53-4069-972f-7a362554ba3f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752548} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0231cd5 a1c82f79-2c53-4069-972f-7a362554ba3f /tidb/cdc/default/default/upstream/7489360124401974112 {"id":7489360124401974112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x pass test: move table [Fri Apr 4 15:42:31 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.51455147.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_stop_delay.t not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360153007907979 TiDB Cluster ID. tikv_gc_leader_uuid 657ff015d60000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:4390, start at 2025-04-04 15:42:30.249832528 +0800 CST m=+2.296752887 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:30.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:30.232 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:30.232 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360153007907979 TiDB Cluster ID. tikv_gc_leader_uuid 657ff015d60000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:4390, start at 2025-04-04 15:42:30.249832528 +0800 CST m=+2.296752887 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:30.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:30.232 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:30.232 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360142444855770 TiDB Cluster ID. tikv_gc_leader_uuid 657ff015f7c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:4466, start at 2025-04-04 15:42:30.407922743 +0800 CST m=+2.361141757 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:30.415 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:30.417 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:30.417 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } }lease 223195ffbfe34612 revoked check diff successfully wait process cdc.test exit for 1-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 6.93 secs (581207488 bytes/sec) [Pipeline] { [Pipeline] cache pass test: set log level check diff successfully wait process cdc.test exit for 2-th time... wait process 5267 exit for 1-th time... table processor_stop_delay.t exists wait process cdc.test exit for 3-th time... wait process 5267 exit for 2-th time... check diff failed 1-th time, retry later pass test: remove changefeed [Fri Apr 4 15:42:34 CST 2025] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.57715773.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:42:34 CST 2025] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5267) - No such process wait process 5267 exit for 3-th time... process 5267 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:42:34 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.54655467.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:34 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e858a41-10e5-46ba-a3a3-111282440aec {"id":"9e858a41-10e5-46ba-a3a3-111282440aec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02f9ed5 9e858a41-10e5-46ba-a3a3-111282440aec /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e858a41-10e5-46ba-a3a3-111282440aec {"id":"9e858a41-10e5-46ba-a3a3-111282440aec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02f9ed5 9e858a41-10e5-46ba-a3a3-111282440aec /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e858a41-10e5-46ba-a3a3-111282440aec {"id":"9e858a41-10e5-46ba-a3a3-111282440aec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02f9ed5 9e858a41-10e5-46ba-a3a3-111282440aec /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5208.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7489360144171780271,"namespace":"default","id":"ddl-manager","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:42:35.055226218+08:00","start_ts":457114269769793542,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114269769793542,"checkpoint_ts":457114269769793542,"checkpoint_time":"2025-04-04 15:42:34.969"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... pass test: resign owner + set +x pass test: get tso =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: mysql... <<================= check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:37 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d72df37f-f9d2-427f-92d8-e9d469544956 {"id":"d72df37f-f9d2-427f-92d8-e9d469544956","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc03674d6 d72df37f-f9d2-427f-92d8-e9d469544956 /tidb/cdc/default/default/upstream/7489360153007907979 {"id":7489360153007907979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d72df37f-f9d2-427f-92d8-e9d469544956 {"id":"d72df37f-f9d2-427f-92d8-e9d469544956","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc03674d6 d72df37f-f9d2-427f-92d8-e9d469544956 /tidb/cdc/default/default/upstream/7489360153007907979 {"id":7489360153007907979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d72df37f-f9d2-427f-92d8-e9d469544956 {"id":"d72df37f-f9d2-427f-92d8-e9d469544956","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc03674d6 d72df37f-f9d2-427f-92d8-e9d469544956 /tidb/cdc/default/default/upstream/7489360153007907979 {"id":7489360153007907979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x pass test: verify table + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:37 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/44410031-8e7c-4899-bc2e-7114410cc2f4 {UpstreamID:7489360112277712112 Namespace:default ID:44410031-8e7c-4899-bc2e-7114410cc2f4 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:42:28.601313711 +0800 CST StartTs:457114268080275460 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039d5ef0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114268093382662} {CheckpointTs:457114268984672265 MinTableBarrierTs:457114270295130124 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268984672265, checkpointTs: 457114268984672265, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec73b812-c5be-4044-9998-dbad35de04df {"id":"ec73b812-c5be-4044-9998-dbad35de04df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b2177 ec73b812-c5be-4044-9998-dbad35de04df /tidb/cdc/default/default/changefeed/info/44410031-8e7c-4899-bc2e-7114410cc2f4 {"upstream-id":7489360112277712112,"namespace":"default","changefeed-id":"44410031-8e7c-4899-bc2e-7114410cc2f4","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:42:28.601313711+08:00","start-ts":457114268080275460,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114268093382662} /tidb/cdc/default/default/changefeed/status/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":457114268984672265,"min-table-barrier-ts":457114270295130124,"admin-job-type":0} /tidb/cdc/default/default/task/position/ec73b812-c5be-4044-9998-dbad35de04df/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/44410031-8e7c-4899-bc2e-7114410cc2f4 {UpstreamID:7489360112277712112 Namespace:default ID:44410031-8e7c-4899-bc2e-7114410cc2f4 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:42:28.601313711 +0800 CST StartTs:457114268080275460 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039d5ef0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114268093382662} {CheckpointTs:457114268984672265 MinTableBarrierTs:457114270295130124 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268984672265, checkpointTs: 457114268984672265, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec73b812-c5be-4044-9998-dbad35de04df {"id":"ec73b812-c5be-4044-9998-dbad35de04df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b2177 ec73b812-c5be-4044-9998-dbad35de04df /tidb/cdc/default/default/changefeed/info/44410031-8e7c-4899-bc2e-7114410cc2f4 {"upstream-id":7489360112277712112,"namespace":"default","changefeed-id":"44410031-8e7c-4899-bc2e-7114410cc2f4","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:42:28.601313711+08:00","start-ts":457114268080275460,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114268093382662} /tidb/cdc/default/default/changefeed/status/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":457114268984672265,"min-table-barrier-ts":457114270295130124,"admin-job-type":0} /tidb/cdc/default/default/task/position/ec73b812-c5be-4044-9998-dbad35de04df/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/44410031-8e7c-4899-bc2e-7114410cc2f4 {UpstreamID:7489360112277712112 Namespace:default ID:44410031-8e7c-4899-bc2e-7114410cc2f4 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:42:28.601313711 +0800 CST StartTs:457114268080275460 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039d5ef0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114268093382662} {CheckpointTs:457114268984672265 MinTableBarrierTs:457114270295130124 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268984672265, checkpointTs: 457114268984672265, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec73b812-c5be-4044-9998-dbad35de04df {"id":"ec73b812-c5be-4044-9998-dbad35de04df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752554} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc01b2177 ec73b812-c5be-4044-9998-dbad35de04df /tidb/cdc/default/default/changefeed/info/44410031-8e7c-4899-bc2e-7114410cc2f4 {"upstream-id":7489360112277712112,"namespace":"default","changefeed-id":"44410031-8e7c-4899-bc2e-7114410cc2f4","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:42:28.601313711+08:00","start-ts":457114268080275460,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114268093382662} /tidb/cdc/default/default/changefeed/status/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":457114268984672265,"min-table-barrier-ts":457114270295130124,"admin-job-type":0} /tidb/cdc/default/default/task/position/ec73b812-c5be-4044-9998-dbad35de04df/44410031-8e7c-4899-bc2e-7114410cc2f4 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360112277712112 {"id":7489360112277712112,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id check diff failed 1-th time, retry later "id": "ec73b812-c5be-4044-9998-dbad35de04df", "cluster-id": "default" run task successfully pass test: create changefeed v2 The 1 times to try to start tidb cluster... check diff failed 2-th time, retry later table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later capture_id: ec73b812-c5be-4044-9998-dbad35de04df check_processor_table_count http://127.0.0.1:2379 44410031-8e7c-4899-bc2e-7114410cc2f4 ec73b812-c5be-4044-9998-dbad35de04df 1 run task successfully check_processor_table_count http://127.0.0.1:2379 44410031-8e7c-4899-bc2e-7114410cc2f4 ec73b812-c5be-4044-9998-dbad35de04df 0 pass test: remove changefeed v2 table count 1 does equal to expected count 0 run task failed 1-th time, retry later table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later check diff failed 3-th time, retry later status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock check_processor_table_count http://127.0.0.1:2379 44410031-8e7c-4899-bc2e-7114410cc2f4 ec73b812-c5be-4044-9998-dbad35de04df 0 run task successfully ***************** properties ***************** "operationcount"="0" "mysql.db"="changefeed_reconstruct" "requestdistribution"="uniform" "dotransactions"="false" "mysql.port"="4000" "threadcount"="4" "mysql.user"="root" "readproportion"="0" "recordcount"="50" "updateproportion"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "workload"="core" "insertproportion"="0" "scanproportion"="0" ********************************************** Run finished, takes 21.84502ms INSERT - Takes(s): 0.0, Count: 48, OPS: 2631.3, Avg(us): 1608, Min(us): 1026, Max(us): 4757, 95th(us): 4000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.26 secs (487685667 bytes/sec) [Pipeline] { [Pipeline] cache wait process cdc.test exit for 1-th time... table changefeed_pause_resume.t3 exists start tidb cluster in /tmp/tidb_cdc_test/cdc Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:42:45 CST 2025] <<<<<< run test case http_api_tls success! >>>>>> Verifying downstream PD is started... check diff failed 5-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/syncpoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff successfully check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:49 CST 2025] <<<<<< run test case changefeed_reconstruct success! >>>>>> wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:49 CST 2025] <<<<<< run test case processor_stop_delay success! >>>>>> check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 4 15:42:48 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ true != \n\o ]] + set +x + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.53135315.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 3-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later wait process 5150 exit for 1-th time... wait process 5150 exit for 2-th time... wait process 5150 exit for 3-th time... wait process 5150 exit for 4-th time... wait process 5150 exit for 5-th time... wait process 5150 exit for 6-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360236341715793 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02c02c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:4023, start at 2025-04-04 15:42:52.943378114 +0800 CST m=+2.677712179 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.952 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360236341715793 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02c02c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:4023, start at 2025-04-04 15:42:52.943378114 +0800 CST m=+2.677712179 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.952 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360248232048906 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02c1c4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:4097, start at 2025-04-04 15:42:53.061108185 +0800 CST m=+2.693957745 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:53.069 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:53.041 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:53.041 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.52 secs (473186462 bytes/sec) [Pipeline] { wait process 5150 exit for 7-th time... [Pipeline] cache wait process 5150 exit for 8-th time... wait process 5150 exit for 9-th time... table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:54 CST 2025] <<<<<< run test case sink_retry success! >>>>>> check diff failed 5-th time, retry later wait process 5150 exit for 10-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360256380344705 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02dda80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:9204, start at 2025-04-04 15:42:54.834617579 +0800 CST m=+2.335551896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:54.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:54.826 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:54.826 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360256380344705 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02dda80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:9204, start at 2025-04-04 15:42:54.834617579 +0800 CST m=+2.335551896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:54.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:54.826 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:54.826 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5150) - No such process wait process 5150 exit for 11-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360254040359090 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02e0c40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:9275, start at 2025-04-04 15:42:55.041302644 +0800 CST m=+2.470918129 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:55.050 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:55.025 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:55.025 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint/tiflash/log/error.log process 5150 already exit [Fri Apr 4 15:42:56 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53095311.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:42:56 CST 2025] <<<<<< START cdc server in cdc case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.54675469.out server --log-file /tmp/tidb_cdc_test/cdc/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.10510.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:59 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8afa134-6363-487e-b9a9-e31fa1859aac {"id":"d8afa134-6363-487e-b9a9-e31fa1859aac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08d73e1 d8afa134-6363-487e-b9a9-e31fa1859aac /tidb/cdc/default/default/upstream/7489360236341715793 {"id":7489360236341715793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8afa134-6363-487e-b9a9-e31fa1859aac {"id":"d8afa134-6363-487e-b9a9-e31fa1859aac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08d73e1 d8afa134-6363-487e-b9a9-e31fa1859aac /tidb/cdc/default/default/upstream/7489360236341715793 {"id":7489360236341715793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8afa134-6363-487e-b9a9-e31fa1859aac {"id":"d8afa134-6363-487e-b9a9-e31fa1859aac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08d73e1 d8afa134-6363-487e-b9a9-e31fa1859aac /tidb/cdc/default/default/upstream/7489360236341715793 {"id":7489360236341715793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.cli.5520.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /tmp/tidb_cdc_test/cdc/pulsar_test.toml Create changefeed successfully! ID: 44606c96-43fa-4aaf-9055-baccc713d958 Info: {"upstream_id":7489360236341715793,"namespace":"default","id":"44606c96-43fa-4aaf-9055-baccc713d958","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:00.272869004+08:00","start_ts":457114276367958022,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114276367958022,"checkpoint_ts":457114276367958022,"checkpoint_time":"2025-04-04 15:43:00.139"} PASS + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114276168204289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114276168204289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:00 CST 2025] <<<<<< START cdc server in syncpoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/DDLSinkInitializeSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.1054610548.out server --log-file /tmp/tidb_cdc_test/syncpoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/api_v2 Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x go: downloading go1.23.4 (linux/amd64) start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d9b1a3a-13de-4e75-aaa6-38350a37d22b {"id":"2d9b1a3a-13de-4e75-aaa6-38350a37d22b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752581} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc09a48ed 2d9b1a3a-13de-4e75-aaa6-38350a37d22b /tidb/cdc/default/default/upstream/7489360256380344705 {"id":7489360256380344705,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d9b1a3a-13de-4e75-aaa6-38350a37d22b {"id":"2d9b1a3a-13de-4e75-aaa6-38350a37d22b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752581} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc09a48ed 2d9b1a3a-13de-4e75-aaa6-38350a37d22b /tidb/cdc/default/default/upstream/7489360256380344705 {"id":7489360256380344705,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d9b1a3a-13de-4e75-aaa6-38350a37d22b {"id":"2d9b1a3a-13de-4e75-aaa6-38350a37d22b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752581} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc09a48ed 2d9b1a3a-13de-4e75-aaa6-38350a37d22b /tidb/cdc/default/default/upstream/7489360256380344705 {"id":7489360256380344705,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.10600.out cli changefeed create --start-ts=457114276168204289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint/conf/changefeed.toml check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Create changefeed successfully! ID: 2a9ac31f-5389-4eaf-8126-0125210c25bb Info: {"upstream_id":7489360256380344705,"namespace":"default","id":"2a9ac31f-5389-4eaf-8126-0125210c25bb","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:04.343688948+08:00","start_ts":457114276168204289,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114276168204289,"checkpoint_ts":457114276168204289,"checkpoint_time":"2025-04-04 15:42:59.377"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3696 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270438260751,"min-table-barrier-ts":457114270438260751,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270438260751,"min-table-barrier-ts":457114270438260751,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270438260751,"min-table-barrier-ts":457114270438260751,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:04 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53705372.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 8.12 secs (496259717 bytes/sec) [Pipeline] { [Pipeline] cache < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:04 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3696 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270569332753,"min-table-barrier-ts":457114270569332753,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270569332753,"min-table-barrier-ts":457114270569332753,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360144171780271 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:35.055226218 +0800 CST StartTs:457114269769793542 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ddd050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114269783162886} {CheckpointTs:457114270438260751 MinTableBarrierTs:457114270438260751 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9746097-32a6-4f55-937e-f1eccc4811e6 {"id":"a9746097-32a6-4f55-937e-f1eccc4811e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02fa116 a9746097-32a6-4f55-937e-f1eccc4811e6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360144171780271,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:42:35.055226218+08:00","start-ts":457114269769793542,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114269783162886} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114270569332753,"min-table-barrier-ts":457114270569332753,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9746097-32a6-4f55-937e-f1eccc4811e6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360144171780271 {"id":7489360144171780271,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x ***************** properties ***************** "mysql.port"="4000" "operationcount"="0" "mysql.host"="127.0.0.1" "insertproportion"="0" "readallfields"="true" "threadcount"="2" "updateproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "recordcount"="10" "workload"="core" "dotransactions"="false" "readproportion"="0" "mysql.db"="testSync" ********************************************** Run finished, takes 24.509224ms INSERT - Takes(s): 0.0, Count: 9, OPS: 1351.0, Avg(us): 3379, Min(us): 1292, Max(us): 17740, 95th(us): 18000, 99th(us): 18000 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore check diff failed 3-th time, retry later Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 2-th check, retry later check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 4 15:43:08 CST 2025] <<<<<< run all test cases cdc_server_tips success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.21.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/google/uuid v1.6.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading golang.org/x/net v0.33.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading google.golang.org/protobuf v1.35.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading go.uber.org/mock v0.4.0 go: downloading golang.org/x/tools v0.28.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc check diff successfully go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading github.com/robfig/cron v1.2.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/rivo/uniseg v0.4.7 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360319929873674 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03b76c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:6690, start at 2025-04-04 15:43:08.793704699 +0800 CST m=+2.266983848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:08.801 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:08.763 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:08.763 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360319929873674 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03b76c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:6690, start at 2025-04-04 15:43:08.793704699 +0800 CST m=+2.266983848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:08.801 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:08.763 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:08.763 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360318935888145 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03b748000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:6763, start at 2025-04-04 15:43:08.782862815 +0800 CST m=+2.187949965 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:08.791 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:08.754 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:08.754 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 4-th check, retry later go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360319884375937 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03cb300003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:7317, start at 2025-04-04 15:43:10.03072403 +0800 CST m=+3.035011800 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.028 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.028 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360319884375937 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03cb300003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:7317, start at 2025-04-04 15:43:10.03072403 +0800 CST m=+3.035011800 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.028 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.028 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360319947095231 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03cbe40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:7391, start at 2025-04-04 15:43:10.074259752 +0800 CST m=+2.999929578 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.083 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.073 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.073 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/api_v2/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/api_v2/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360325461783555 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03d35c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:9135, start at 2025-04-04 15:43:10.56353966 +0800 CST m=+2.107202625 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.571 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.551 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.551 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360325461783555 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03d35c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:9135, start at 2025-04-04 15:43:10.56353966 +0800 CST m=+2.107202625 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.571 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.551 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.551 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360321931083556 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03d5b0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-0vmr4-q18gq-3c3cn, pid:9212, start at 2025-04-04 15:43:10.748010919 +0800 CST m=+2.195287345 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:10.755 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:10.750 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:10.750 +0800 All versions after safe point can be accessed. (DO NOT EDIT) go: downloading github.com/klauspost/cpuid/v2 v2.2.4 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8060.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 5-th check, retry later [Fri Apr 4 15:43:13 CST 2025] <<<<<< START cdc server in api_v2 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.api_v2.87558757.out server --log-file /tmp/tidb_cdc_test/api_v2/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/api_v2/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.44 secs (541777493 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 1-th time, retry later + set +x + tso='457114279808073729 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114279808073729 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "mysql.host"="127.0.0.1" "recordcount"="10" "operationcount"="0" "updateproportion"="0" "mysql.user"="root" "mysql.db"="multi_capture_1" "workload"="core" "readproportion"="0" "requestdistribution"="uniform" "threadcount"="2" "readallfields"="true" "scanproportion"="0" "dotransactions"="false" "mysql.port"="4000" "insertproportion"="0" ********************************************** Run finished, takes 8.635126ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1792.0, Avg(us): 1651, Min(us): 1008, Max(us): 2993, 95th(us): 3000, 99th(us): 3000 ***************** properties ***************** "insertproportion"="0" "workload"="core" "threadcount"="2" "mysql.user"="root" "mysql.host"="127.0.0.1" "readallfields"="true" "operationcount"="0" "scanproportion"="0" "mysql.db"="multi_capture_2" "updateproportion"="0" "dotransactions"="false" "mysql.port"="4000" "recordcount"="10" "requestdistribution"="uniform" "readproportion"="0" ********************************************** Run finished, takes 9.229542ms INSERT - Takes(s): 0.0, Count: 9, OPS: 1642.6, Avg(us): 1565, Min(us): 1140, Max(us): 3654, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "insertproportion"="0" "scanproportion"="0" "mysql.user"="root" "readallfields"="true" "workload"="core" "mysql.db"="multi_capture_3" "mysql.port"="4000" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "operationcount"="0" "readproportion"="0" "dotransactions"="false" "recordcount"="10" "updateproportion"="0" "threadcount"="2" ********************************************** Run finished, takes 9.275012ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1813.3, Avg(us): 1793, Min(us): 1024, Max(us): 3964, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "operationcount"="0" "readproportion"="0" "updateproportion"="0" "threadcount"="2" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "recordcount"="10" "readallfields"="true" "insertproportion"="0" "dotransactions"="false" "scanproportion"="0" "mysql.db"="multi_capture_4" "mysql.user"="root" "workload"="core" "mysql.port"="4000" ********************************************** Run finished, takes 10.846901ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1869.3, Avg(us): 2064, Min(us): 1074, Max(us): 5468, 95th(us): 6000, 99th(us): 6000 table ddl_manager.finish_mark not exists for 6-th check, retry later [Fri Apr 4 15:43:15 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.81648166.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10515.out cli tso query --pd=http://127.0.0.1:2379 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/jmespath/go-jmespath v0.4.0 + set +x + tso='457114280329478146 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114280329478146 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:16 CST 2025] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1055110553.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 7-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:16 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/99bc7fbe-8e7b-4376-b00f-ea619bc37f72 {"id":"99bc7fbe-8e7b-4376-b00f-ea619bc37f72","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752593} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cef8e2 99bc7fbe-8e7b-4376-b00f-ea619bc37f72 /tidb/cdc/default/default/upstream/7489360319884375937 {"id":7489360319884375937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/99bc7fbe-8e7b-4376-b00f-ea619bc37f72 {"id":"99bc7fbe-8e7b-4376-b00f-ea619bc37f72","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752593} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cef8e2 99bc7fbe-8e7b-4376-b00f-ea619bc37f72 /tidb/cdc/default/default/upstream/7489360319884375937 {"id":7489360319884375937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/99bc7fbe-8e7b-4376-b00f-ea619bc37f72 {"id":"99bc7fbe-8e7b-4376-b00f-ea619bc37f72","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752593} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cef8e2 99bc7fbe-8e7b-4376-b00f-ea619bc37f72 /tidb/cdc/default/default/upstream/7489360319884375937 {"id":7489360319884375937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cf-blackhole Info: {"upstream_id":7489360319884375937,"namespace":"default","id":"cf-blackhole","sink_uri":"blackhole:","create_time":"2025-04-04T15:43:16.802768293+08:00","start_ts":457114280703557637,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114280703557637,"checkpoint_ts":457114280703557637,"checkpoint_time":"2025-04-04 15:43:16.678"} + endpoints=http://127.0.0.1:2379 + changefeed_id=cf-blackhole + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cf-blackhole -s + info='{ "upstream_id": 7489360319884375937, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114280703557637, "checkpoint_time": "2025-04-04 15:43:16.678", "error": null }' + echo '{ "upstream_id": 7489360319884375937, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114280703557637, "checkpoint_time": "2025-04-04 15:43:16.678", "error": null }' { "upstream_id": 7489360319884375937, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114280703557637, "checkpoint_time": "2025-04-04 15:43:16.678", "error": null } ++ echo '{' '"upstream_id":' 7489360319884375937, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 457114280703557637, '"checkpoint_time":' '"2025-04-04' '15:43:16.678",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360319884375937, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 457114280703557637, '"checkpoint_time":' '"2025-04-04' '15:43:16.678",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + GO111MODULE=on + go run main.go model.go request.go cases.go go: downloading go1.23.4 (linux/amd64) check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360347588497912 TiDB Cluster ID. tikv_gc_leader_uuid 657ff042af4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:3873, start at 2025-04-04 15:43:16.203734503 +0800 CST m=+2.464778455 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:16.211 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:16.207 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:16.207 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360347588497912 TiDB Cluster ID. tikv_gc_leader_uuid 657ff042af4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:3873, start at 2025-04-04 15:43:16.203734503 +0800 CST m=+2.464778455 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:16.211 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:16.207 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:16.207 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360340907261018 TiDB Cluster ID. tikv_gc_leader_uuid 657ff042d40000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:3941, start at 2025-04-04 15:43:16.32195115 +0800 CST m=+2.495111564 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:16.331 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:16.304 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:16.304 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:18 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:18 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.82178219.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_ddl/sync_diff/output" source-instances = ["tidb"] target-instance = "mysql" target-check-tables = ["consistent_replicate_ddl.usertable*","consistent_replicate_ddl.check*"] [data-sources] [data-sources.tidb] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "457114248335065097" [data-sources.mysql] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:43:15.496 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:43:15.497 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:43:15.497 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:43:15.497 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:43:15.498 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114248138457089] [resolvedTs=457114248335065097] [2025/04/04 15:43:15.499 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:43:15.499 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:43:15.499 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:43:15.499 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.128976ms] [2025/04/04 15:43:15.501 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=3.277441ms] [2025/04/04 15:43:15.504 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:43:15.507 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:43:15.509 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:43:15.509 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:43:15.509 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:43:15.509 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:43:15.514 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:43:15.517 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:43:15.517 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:43:15.517 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248125349925,\"CommitTs\":457114248138457089,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1\",\"TableID\":116,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:15.518 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:15.518 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248138457089] [DDL="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [2025/04/04 15:43:15.587 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=68.412472ms] [sql="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [2025/04/04 15:43:15.587 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248151826445,\"CommitTs\":457114248151826448,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1\",\"TableID\":122,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:15.587 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:15.587 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248151826448] [DDL="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [2025/04/04 15:43:15.626 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=38.406053ms] [sql="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [2025/04/04 15:43:15.626 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248164671523,\"CommitTs\":457114248177778689,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1_1\",\"TableID\":122,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:15.827 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:15.827 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248177778689] [DDL="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [2025/04/04 15:43:15.875 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=47.48759ms] [sql="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [2025/04/04 15:43:15.875 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248190885904,\"CommitTs\":457114248190885908,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl1\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.076 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248190885908] [DDL="CREATE DATABASE consistent_replicate_ddl1"] [2025/04/04 15:43:16.098 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=21.217421ms] [sql="CREATE DATABASE consistent_replicate_ddl1"] [2025/04/04 15:43:16.098 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248203993103,\"CommitTs\":457114248203993106,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.098 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:16.098 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248203993106] [DDL="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:43:16.128 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=29.30218ms] [sql="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:43:16.128 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248217100290,\"CommitTs\":457114248217100294,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.128 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:16.128 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248217100294] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:43:16.158 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=30.056882ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:43:16.159 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248243314703,\"CommitTs\":457114248243314708,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.359 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248243314708] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [2025/04/04 15:43:16.418 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=59.310495ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [2025/04/04 15:43:16.419 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248256421938,\"CommitTs\":457114248256421942,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.419 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248256421942] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [2025/04/04 15:43:16.483 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=64.382395ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [2025/04/04 15:43:16.483 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248269529131,\"CommitTs\":457114248269529135,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl1\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.483 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248269529135] [DDL="DROP DATABASE consistent_replicate_ddl1"] [2025/04/04 15:43:16.529 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=46.095572ms] [sql="DROP DATABASE consistent_replicate_ddl1"] [2025/04/04 15:43:16.529 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248282636321,\"CommitTs\":457114248282636325,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.530 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:16.530 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248282636325] [DDL="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [2025/04/04 15:43:16.558 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=27.535497ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [2025/04/04 15:43:16.558 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248295743524,\"CommitTs\":457114248295743527,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2_1\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.558 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:16.558 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248295743527] [DDL="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [2025/04/04 15:43:16.595 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=37.019051ms] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [2025/04/04 15:43:16.595 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248295743524,\"CommitTs\":457114248295743527,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3_1\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.596 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:43:16.596 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248295743527] [DDL="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [2025/04/04 15:43:16.636 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=40.215893ms] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [2025/04/04 15:43:16.636 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248308850714,\"CommitTs\":457114248308850718,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3_1\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.636 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248308850718] [DDL="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [2025/04/04 15:43:16.664 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=27.385793ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [2025/04/04 15:43:16.664 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114248335065093,\"CommitTs\":457114248335065097,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"check1\",\"TableID\":137,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:43:16.865 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114248335065097] [DDL="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [2025/04/04 15:43:16.890 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=24.748116ms] [sql="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [2025/04/04 15:43:16.890 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:16.890 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:16.990 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:16.990 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:17.091 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:17.091 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114248335065097] [2025/04/04 15:43:17.191 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=101] [appliedDDLCount=14] [currentCheckpoint=457114248335065097] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:43:17.191 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] Apply redo log successfully table consistent_replicate_ddl.check1 exists check diff successfully [Fri Apr 4 15:43:17 CST 2025] <<<<<< run test case consistent_replicate_ddl success! >>>>>> count(*) 50 check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 8-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8b096b43-dd9b-4d05-b98f-3dc160ed2dbd {"id":"8b096b43-dd9b-4d05-b98f-3dc160ed2dbd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752597} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d87fe5 8b096b43-dd9b-4d05-b98f-3dc160ed2dbd /tidb/cdc/default/default/upstream/7489360325461783555 {"id":7489360325461783555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8b096b43-dd9b-4d05-b98f-3dc160ed2dbd {"id":"8b096b43-dd9b-4d05-b98f-3dc160ed2dbd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752597} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d87fe5 8b096b43-dd9b-4d05-b98f-3dc160ed2dbd /tidb/cdc/default/default/upstream/7489360325461783555 {"id":7489360325461783555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8b096b43-dd9b-4d05-b98f-3dc160ed2dbd {"id":"8b096b43-dd9b-4d05-b98f-3dc160ed2dbd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752597} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d87fe5 8b096b43-dd9b-4d05-b98f-3dc160ed2dbd /tidb/cdc/default/default/upstream/7489360325461783555 {"id":7489360325461783555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10606.out cli changefeed create --start-ts=457114280329478146 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 07536c11-609f-4a88-8c06-2a4c4f025825 Info: {"upstream_id":7489360325461783555,"namespace":"default","id":"07536c11-609f-4a88-8c06-2a4c4f025825","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:20.319497167+08:00","start_ts":457114280329478146,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114280329478146,"checkpoint_ts":457114280329478146,"checkpoint_time":"2025-04-04 15:43:15.251"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5290.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 9-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:21 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:21 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.82698271.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8303 failed: Connection refused * Failed to connect to 127.0.0.1 port 8303: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + set +x + tso='457114281707307009 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114281707307009 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "scanproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "workload"="core" "dotransactions"="false" "readproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "threadcount"="4" "mysql.user"="root" "readallfields"="true" "recordcount"="20" "mysql.db"="changefeed_error" "updateproportion"="0" ********************************************** Run finished, takes 10.098278ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2957.0, Avg(us): 1897, Min(us): 1088, Max(us): 3658, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:43:22 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.53505352.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading google.golang.org/grpc v1.64.0 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/xdg/scram v1.0.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/IBM/sarama v1.41.2 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading go.uber.org/multierr v1.11.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/google/btree v1.1.2 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 table ddl_manager.finish_mark not exists for 10-th check, retry later go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/go-playground/validator/v10 v10.14.0 check diff failed 5-th time, retry later go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.35.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc) 4030545920 bytes in 7.40 secs (544813185 bytes/sec) [Pipeline] { [Pipeline] sh go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/eapache/go-resiliency v1.4.0 start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8303 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:24 GMT < Content-Length: 1735 < Content-Type: text/plain; charset=utf-8 < { [1735 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11ff303a-ccfd-4b64-8482-ab542df72276 {"id":"11ff303a-ccfd-4b64-8482-ab542df72276","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3137b 11ff303a-ccfd-4b64-8482-ab542df72276 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11ff303a-ccfd-4b64-8482-ab542df72276 {"id":"11ff303a-ccfd-4b64-8482-ab542df72276","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3137b 11ff303a-ccfd-4b64-8482-ab542df72276 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11ff303a-ccfd-4b64-8482-ab542df72276 {"id":"11ff303a-ccfd-4b64-8482-ab542df72276","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/capture/1e608ec8-2530-4454-97bc-58b065cad932 {"id":"1e608ec8-2530-4454-97bc-58b065cad932","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752598} /tidb/cdc/default/__cdc_meta__/capture/d09b91ef-974c-432c-860d-8f8f5b70c3b6 {"id":"d09b91ef-974c-432c-860d-8f8f5b70c3b6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752595} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3131f d09b91ef-974c-432c-860d-8f8f5b70c3b6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d31357 1e608ec8-2530-4454-97bc-58b065cad932 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0d3137b 11ff303a-ccfd-4b64-8482-ab542df72276 /tidb/cdc/default/default/upstream/7489360319929873674 {"id":7489360319929873674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8319.out cli changefeed create --start-ts=457114279808073729 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 [Pipeline] sh go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G17 [Pipeline] sh go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 Create changefeed successfully! ID: 359ff397-98f2-4632-a7e5-e0cd3de7b619 Info: {"upstream_id":7489360319929873674,"namespace":"default","id":"359ff397-98f2-4632-a7e5-e0cd3de7b619","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:24.639998612+08:00","start_ts":457114279808073729,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114279808073729,"checkpoint_ts":457114279808073729,"checkpoint_time":"2025-04-04 15:43:13.262"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Run cases: clustered_index processor_resolved_ts_fallback GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-htn1z HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=7375a0fc-2191-4d84-ac41-a65bd3f4e267 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G17 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-htn1z pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G11 [Pipeline] sh go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 Run cases: resolve_lock move_table autorandom generate_column GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-1hc17 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=c0fadd3b-7c1f-4b0d-a26a-fdc25a6cbe90 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G11 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl pingcap_tiflow_pull_cdc_integration_test_3828-1hc17 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G20 [Pipeline] sh go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c table ddl_manager.finish_mark not exists for 11-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:25 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c {"id":"220a7be9-951a-47f2-a3c6-9c72c1070a0c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752602} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e951f3 220a7be9-951a-47f2-a3c6-9c72c1070a0c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c {"id":"220a7be9-951a-47f2-a3c6-9c72c1070a0c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752602} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e951f3 220a7be9-951a-47f2-a3c6-9c72c1070a0c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c {"id":"220a7be9-951a-47f2-a3c6-9c72c1070a0c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752602} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e951f3 220a7be9-951a-47f2-a3c6-9c72c1070a0c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5409.out cli changefeed create --start-ts=457114281707307009 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error Run cases: tidb_mysql_test ddl_reentrant multi_cdc_cluster GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-r1g2f HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=476d9149-3148-4823-8875-1bec0e927dfb GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G20 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k pingcap_tiflow_pull_cdc_integration_test_3828-r1g2f PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/tidb_mysql_test/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G21 [Pipeline] sh go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7489360347588497912,"namespace":"default","id":"changefeed-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:25.571065073+08:00","start_ts":457114281707307009,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009,"checkpoint_time":"2025-04-04 15:43:20.507"} PASS Run cases: bank kill_owner_with_ddl owner_remove_table_error GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-7b293 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=4e862ac6-6569-4e2e-aecd-71638a244c60 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G21 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-7b293 pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G15 [Pipeline] sh coverage: 2.4% of statements in github.com/pingcap/tiflow/... Run cases: new_ci_collation batch_add_table multi_rocks ci_collation_compatibility GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-k1gsf HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=6b7e1446-2426-4f74-be9f-1f148dfa547d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G15 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-k1gsf pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G13 [Pipeline] sh go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 Run cases: tiflash region_merge common_1 GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-9jjm4 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=54e67390-e181-4e03-b25c-afe74689f11f GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G13 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q pingcap_tiflow_pull_cdc_integration_test_3828-9jjm4 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x table multi_capture_1.usertable not exists for 1-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G16 [Pipeline] sh go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/DataDog/zstd v1.5.5 Run cases: owner_resign processor_etcd_worker_delay sink_hang GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-1z1c1 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=f74fb1a0-1861-4b33-b1d2-32a5bb4ae8d4 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G16 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-1z1c1 pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G12 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/xdg-go/scram v1.1.2 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G19 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-9ffrv HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=fc490852-d320-43f9-b696-ad9e90c45e1f GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G12 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-9ffrv pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: mysql... <<================= go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 [Pipeline] sh Run cases: changefeed_fast_fail batch_update_to_no_batch changefeed_resume_with_checkpoint_ts GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-xb2bm HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=659ce761-4863-4466-a890-d11c4a1c7e3e GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G19 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-xb2bm pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_fast_fail/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G14 table ddl_manager.finish_mark not exists for 12-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:43:25.682085471+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at 457114281707307009" } }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:43:25.682085471+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at 457114281707307009" } }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:43:25.682085471+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at 457114281707307009" } } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114281707307009, '"checkpoint_time":' '"2025-04-04' '15:43:20.507",' '"error":' '{' '"time":' '"2025-04-04T15:43:25.682085471+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at '457114281707307009"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114281707307009, '"checkpoint_time":' '"2025-04-04' '15:43:20.507",' '"error":' '{' '"time":' '"2025-04-04T15:43:25.682085471+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at '457114281707307009"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at 457114281707307009' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114281707306709 is earlier than or equal to GC safepoint at 457114281707307009 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5479.out cli changefeed resume -c changefeed-error Run cases: changefeed_finish force_replicate_table + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G18 GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-9cfmv HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=ab9ca0eb-b01d-4d47-ad14-e4832804d263 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G14 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3828-9cfmv pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Run cases: availability http_proxies sequence GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908057860688842754","prowjobid":"b24ff031-df4c-4fa7-8bfb-89730f6d23f0","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860688842754 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3828-87zdl HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=changes JENKINS_NODE_COOKIE=44815f12-8d53-46e2-8bd7-253db45cb648 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3828 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=b24ff031-df4c-4fa7-8bfb-89730f6d23f0 TEST_GROUP=G18 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3828/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3828 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3828 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40 pingcap_tiflow_pull_cdc_integration_test_3828-87zdl PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/availability/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide The 1 times to try to start tidb cluster... + set +x table changefeed_error.usertable exists check diff failed 1-th time, retry later * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DC261DFAF1B < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:43:29 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... check diff failed 1-th time, retry later go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 table ddl_manager.finish_mark not exists for 13-th check, retry later go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/kr/text v0.2.0 start tidb cluster in /tmp/tidb_cdc_test/bank Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later The 1 times to try to start tidb cluster... go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/dolthub/maphash v0.1.0 start tidb cluster in /tmp/tidb_cdc_test/new_ci_collation Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/clustered_index Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/apache/thrift v0.16.0 go: downloading cloud.google.com/go/kms v1.15.8 start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/rivo/uniseg v0.4.7 table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible start tidb cluster in /tmp/tidb_cdc_test/changefeed_fast_fail Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table partition_table.t2 not exists for 2-th check, retry later go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 14-th check, retry later Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/availability Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 check diff successfully ***************** properties ***************** "updateproportion"="0" "threadcount"="4" "insertproportion"="0" "dotransactions"="false" "workload"="core" "operationcount"="0" "requestdistribution"="uniform" "mysql.port"="4000" "readallfields"="true" "readproportion"="0" "mysql.user"="root" "recordcount"="20" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_error" "scanproportion"="0" ********************************************** Run finished, takes 4.48329ms INSERT - Takes(s): 0.0, Count: 20, OPS: 6527.2, Avg(us): 737, Min(us): 391, Max(us): 1803, 95th(us): 2000, 99th(us): 2000 check diff successfully {"id":"220a7be9-951a-47f2-a3c6-9c72c1070a0c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752602} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + [[ /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading k8s.io/apimachinery v0.29.11 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... table partition_table.t2 not exists for 3-th check, retry later check diff failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 15-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_gbk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + [[ /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360417347197932 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05248c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:6376, start at 2025-04-04 15:43:32.173488194 +0800 CST m=+2.327553344 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.182 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.182 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360417347197932 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05248c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:6376, start at 2025-04-04 15:43:32.173488194 +0800 CST m=+2.327553344 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.182 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.182 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360419136531434 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05279c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:6443, start at 2025-04-04 15:43:32.362606802 +0800 CST m=+2.411631897 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.370 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.377 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.377 +0800 All versions after safe point can be accessed. (DO NOT EDIT) start tidb cluster in /tmp/tidb_cdc_test/many_pk_or_uk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table multi_capture_4.usertable exists check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/tidb_mysql_test Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... check diff failed 4-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore table ddl_manager.finish_mark not exists for 16-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 4-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:43:36 CST 2025] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.76547656.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ***************** properties ***************** "readallfields"="true" "recordcount"="20" "workload"="core" "insertproportion"="0" "mysql.user"="root" "operationcount"="0" "mysql.port"="4000" "readproportion"="0" "scanproportion"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "dotransactions"="false" "threadcount"="2" "mysql.db"="multi_capture_1" "requestdistribution"="uniform" ********************************************** Run finished, takes 11.765425ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1871.9, Avg(us): 1026, Min(us): 483, Max(us): 1816, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.user"="root" "insertproportion"="0" "mysql.port"="4000" "recordcount"="20" "updateproportion"="0" "scanproportion"="0" "operationcount"="0" "readproportion"="0" "mysql.db"="multi_capture_2" "dotransactions"="false" "workload"="core" "threadcount"="2" "readallfields"="true" ********************************************** Run finished, takes 10.670412ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2145.6, Avg(us): 1014, Min(us): 514, Max(us): 1621, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "dotransactions"="false" "threadcount"="2" "workload"="core" "mysql.host"="127.0.0.1" "readallfields"="true" "requestdistribution"="uniform" "updateproportion"="0" "mysql.user"="root" "insertproportion"="0" "recordcount"="20" "operationcount"="0" "readproportion"="0" "mysql.db"="multi_capture_3" "mysql.port"="4000" "scanproportion"="0" ********************************************** Run finished, takes 11.491708ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1925.5, Avg(us): 988, Min(us): 427, Max(us): 1756, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "operationcount"="0" "requestdistribution"="uniform" "threadcount"="2" "scanproportion"="0" "recordcount"="20" "dotransactions"="false" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_4" "readallfields"="true" "insertproportion"="0" "updateproportion"="0" "readproportion"="0" ********************************************** Run finished, takes 10.72315ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2099.0, Avg(us): 1000, Min(us): 423, Max(us): 2132, 95th(us): 3000, 99th(us): 3000 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 17-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 5-th check, retry later check diff failed 5-th time, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + [[ /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/220a7be9-951a-47f2-a3c6-9c72c1070a0c + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360433438143520 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05697c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:1278, start at 2025-04-04 15:43:36.562534817 +0800 CST m=+2.423885758 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.572 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.543 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.543 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360433438143520 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05697c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:1278, start at 2025-04-04 15:43:36.562534817 +0800 CST m=+2.423885758 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.572 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.543 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.543 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360439779863214 TiDB Cluster ID. tikv_gc_leader_uuid 657ff056b080007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:1352, start at 2025-04-04 15:43:36.653267167 +0800 CST m=+2.416452750 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.660 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360437276853351 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0568880039 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:1350, start at 2025-04-04 15:43:36.501919925 +0800 CST m=+2.109276248 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.506 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.506 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360437276853351 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0568880039 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:1350, start at 2025-04-04 15:43:36.501919925 +0800 CST m=+2.109276248 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.506 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.506 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360439085489224 TiDB Cluster ID. tikv_gc_leader_uuid 657ff056b3c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:1424, start at 2025-04-04 15:43:36.702771307 +0800 CST m=+2.204294154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:36.710 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:36.704 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:36.704 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... check diff successfully table ddl_manager.finish_mark not exists for 18-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360442433744372 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0576640003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:1278, start at 2025-04-04 15:43:37.371620331 +0800 CST m=+2.170172052 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360442433744372 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0576640003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:1278, start at 2025-04-04 15:43:37.371620331 +0800 CST m=+2.170172052 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360436066391899 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0578b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:1353, start at 2025-04-04 15:43:37.549016912 +0800 CST m=+2.261631557 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.556 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.518 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.518 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:39 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e43c958d-c431-499e-b236-f59d53769346 {"id":"e43c958d-c431-499e-b236-f59d53769346","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1284ed6 e43c958d-c431-499e-b236-f59d53769346 /tidb/cdc/default/default/upstream/7489360417347197932 {"id":7489360417347197932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e43c958d-c431-499e-b236-f59d53769346 {"id":"e43c958d-c431-499e-b236-f59d53769346","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1284ed6 e43c958d-c431-499e-b236-f59d53769346 /tidb/cdc/default/default/upstream/7489360417347197932 {"id":7489360417347197932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e43c958d-c431-499e-b236-f59d53769346 {"id":"e43c958d-c431-499e-b236-f59d53769346","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1284ed6 e43c958d-c431-499e-b236-f59d53769346 /tidb/cdc/default/default/upstream/7489360417347197932 {"id":7489360417347197932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7712.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml Create changefeed successfully! ID: d0496b07-d0e2-468e-b58e-d22a293aae39 Info: {"upstream_id":7489360417347197932,"namespace":"default","id":"d0496b07-d0e2-468e-b58e-d22a293aae39","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:39.770931774+08:00","start_ts":457114286733918212,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_name*"],"ignore_event":["rename table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114286733918212,"checkpoint_ts":457114286733918212,"checkpoint_time":"2025-04-04 15:43:39.682"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360435876110807 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057f640005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:1357, start at 2025-04-04 15:43:37.95397407 +0800 CST m=+2.406293736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.962 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.945 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.945 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360435876110807 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057f640005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:1357, start at 2025-04-04 15:43:37.95397407 +0800 CST m=+2.406293736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.962 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.945 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.945 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360444094380970 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05802c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:1425, start at 2025-04-04 15:43:38.030256588 +0800 CST m=+2.375117269 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.040 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.045 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.045 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bank/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bank/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bank/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/bank/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/bank/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360432750340396 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057dc8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:1350, start at 2025-04-04 15:43:37.859739931 +0800 CST m=+2.207344001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.867 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360432750340396 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057dc8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:1350, start at 2025-04-04 15:43:37.859739931 +0800 CST m=+2.207344001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.867 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360443838010344 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057f54000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:1423, start at 2025-04-04 15:43:37.970519249 +0800 CST m=+2.231193737 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.978 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.013 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.013 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/clustered_index/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/clustered_index/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360444900490223 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057f94000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:1212, start at 2025-04-04 15:43:37.974126725 +0800 CST m=+2.420825278 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.981 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360444900490223 TiDB Cluster ID. tikv_gc_leader_uuid 657ff057f94000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:1212, start at 2025-04-04 15:43:37.974126725 +0800 CST m=+2.420825278 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:37.981 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:37.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:37.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360434549359508 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058250000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:1284, start at 2025-04-04 15:43:38.174705461 +0800 CST m=+2.548890456 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.182 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.181 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.181 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:40 CST 2025] <<<<<< run test case multi_capture success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360440316725357 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058ef40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:1353, start at 2025-04-04 15:43:38.950796442 +0800 CST m=+2.447231827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.960 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360440316725357 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058ef40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:1353, start at 2025-04-04 15:43:38.950796442 +0800 CST m=+2.447231827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.960 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360447032866970 TiDB Cluster ID. tikv_gc_leader_uuid 657ff059068000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:1428, start at 2025-04-04 15:43:39.073529811 +0800 CST m=+2.489687277 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:39.082 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:39.085 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:39.085 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2656.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/availability/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/availability/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/availability/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/availability/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/availability/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2637.out cli tso query --pd=http://127.0.0.1:2379 table partition_table.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table ddl_manager.finish_mark not exists for 19-th check, retry later table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later [Fri Apr 4 15:43:41 CST 2025] <<<<<< START cdc server in new_ci_collation case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.new_ci_collation.27082710.out server --log-file /tmp/tidb_cdc_test/new_ci_collation/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/new_ci_collation/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360450719664448 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05aa880007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:1340, start at 2025-04-04 15:43:40.747345923 +0800 CST m=+3.457672222 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.756 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.756 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360450719664448 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05aa880007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:1340, start at 2025-04-04 15:43:40.747345923 +0800 CST m=+3.457672222 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.756 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.756 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360448275542460 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05ac640008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:1407, start at 2025-04-04 15:43:40.84456247 +0800 CST m=+3.463414536 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.855 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.855 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.855 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:43:42 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.26372639.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114287038267393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114287038267393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:42 CST 2025] <<<<<< START cdc server in changefeed_fast_fail case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.26772679.out server --log-file /tmp/tidb_cdc_test/changefeed_fast_fail/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_fast_fail/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:43:42 CST 2025] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.26702672.out server --log-file /tmp/tidb_cdc_test/bank/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:43:42 CST 2025] <<<<<< START cdc server in clustered_index case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.clustered_index.26862688.out server --log-file /tmp/tidb_cdc_test/clustered_index/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/clustered_index/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='457114287107211265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114287107211265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:42 CST 2025] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.26902692.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360441792358903 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b2400006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:1367, start at 2025-04-04 15:43:41.21352631 +0800 CST m=+2.545407118 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.222 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.200 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.200 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360441792358903 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b2400006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:1367, start at 2025-04-04 15:43:41.21352631 +0800 CST m=+2.545407118 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.222 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.200 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.200 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360455245197531 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b3b4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:1444, start at 2025-04-04 15:43:41.323153659 +0800 CST m=+2.582260132 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.331 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.293 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.293 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2665.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 20-th check, retry later check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360452397051178 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05bba0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:4169, start at 2025-04-04 15:43:41.820843274 +0800 CST m=+2.603833423 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360452397051178 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05bba0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:4169, start at 2025-04-04 15:43:41.820843274 +0800 CST m=+2.603833423 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360460053703245 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05bdf4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:4244, start at 2025-04-04 15:43:41.971154775 +0800 CST m=+2.640417950 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:42.008 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:42.000 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:42.000 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:43:43 CST 2025] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.26302632.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360450106133187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05c06c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:1401, start at 2025-04-04 15:43:42.107507967 +0800 CST m=+2.391466955 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:42.117 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:42.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:42.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360450106133187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05c06c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:1401, start at 2025-04-04 15:43:42.107507967 +0800 CST m=+2.391466955 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:42.117 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:42.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:42.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360460546794094 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05bfa4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:1464, start at 2025-04-04 15:43:42.102444001 +0800 CST m=+2.306682462 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:42.110 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:42.107 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:42.107 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table event_filter.t1 exists table event_filter.t_normal not exists for 1-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:43:43 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.57115713.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114287679733762 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114287679733762 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:44 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f3aebe9-e252-4631-be22-ebbd8c291cb9 {"id":"0f3aebe9-e252-4631-be22-ebbd8c291cb9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13f1bd9 0f3aebe9-e252-4631-be22-ebbd8c291cb9 /tidb/cdc/default/default/upstream/7489360442433744372 {"id":7489360442433744372,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f3aebe9-e252-4631-be22-ebbd8c291cb9 {"id":"0f3aebe9-e252-4631-be22-ebbd8c291cb9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13f1bd9 0f3aebe9-e252-4631-be22-ebbd8c291cb9 /tidb/cdc/default/default/upstream/7489360442433744372 {"id":7489360442433744372,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f3aebe9-e252-4631-be22-ebbd8c291cb9 {"id":"0f3aebe9-e252-4631-be22-ebbd8c291cb9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13f1bd9 0f3aebe9-e252-4631-be22-ebbd8c291cb9 /tidb/cdc/default/default/upstream/7489360442433744372 {"id":7489360442433744372,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 9ea479a3-9758-4e09-bf11-45630ac01292 Info: {"upstream_id":7489360442433744372,"namespace":"default","id":"9ea479a3-9758-4e09-bf11-45630ac01292","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:43:44.974587784+08:00","start_ts":457114287267905537,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114287267905537,"checkpoint_ts":457114287267905537,"checkpoint_time":"2025-04-04 15:43:41.719"} + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2765.out cli tso query --pd=http://127.0.0.1:2379 run test case test_kill_owner [Fri Apr 4 15:43:45 CST 2025] <<<<<< START cdc server in availability case >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.27052707.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 21-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:45 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38de6e65-67a7-4a37-a0ba-b35d171b1990 {"id":"38de6e65-67a7-4a37-a0ba-b35d171b1990","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13ecdd9 38de6e65-67a7-4a37-a0ba-b35d171b1990 /tidb/cdc/default/default/upstream/7489360432750340396 {"id":7489360432750340396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38de6e65-67a7-4a37-a0ba-b35d171b1990 {"id":"38de6e65-67a7-4a37-a0ba-b35d171b1990","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13ecdd9 38de6e65-67a7-4a37-a0ba-b35d171b1990 /tidb/cdc/default/default/upstream/7489360432750340396 {"id":7489360432750340396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38de6e65-67a7-4a37-a0ba-b35d171b1990 {"id":"38de6e65-67a7-4a37-a0ba-b35d171b1990","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13ecdd9 38de6e65-67a7-4a37-a0ba-b35d171b1990 /tidb/cdc/default/default/upstream/7489360432750340396 {"id":7489360432750340396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 7c303b92-a519-4285-a9de-3b7fbd9f3f0b Info: {"upstream_id":7489360432750340396,"namespace":"default","id":"7c303b92-a519-4285-a9de-3b7fbd9f3f0b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:45.379922999+08:00","start_ts":457114287365685250,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114287365685250,"checkpoint_ts":457114287365685250,"checkpoint_time":"2025-04-04 15:43:42.092"} + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:45 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2728.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 check diff failed 2-th time, retry later table new_ci_collation_test.t1 not exists for 1-th check, retry later Create changefeed successfully! ID: d521d148-2502-4e5d-8467-fd3f88dea099 Info: {"upstream_id":7489360444900490223,"namespace":"default","id":"d521d148-2502-4e5d-8467-fd3f88dea099","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:45.620143427+08:00","start_ts":457114288267984898,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288267984898,"checkpoint_ts":457114288267984898,"checkpoint_time":"2025-04-04 15:43:45.534"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table partition_table.finish_mark not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:45 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b5e749-89f6-4f5d-a177-900ea8fb18a7 {"id":"59b5e749-89f6-4f5d-a177-900ea8fb18a7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc142b6ec 59b5e749-89f6-4f5d-a177-900ea8fb18a7 /tidb/cdc/default/default/upstream/7489360433438143520 {"id":7489360433438143520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b5e749-89f6-4f5d-a177-900ea8fb18a7 {"id":"59b5e749-89f6-4f5d-a177-900ea8fb18a7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc142b6ec 59b5e749-89f6-4f5d-a177-900ea8fb18a7 /tidb/cdc/default/default/upstream/7489360433438143520 {"id":7489360433438143520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b5e749-89f6-4f5d-a177-900ea8fb18a7 {"id":"59b5e749-89f6-4f5d-a177-900ea8fb18a7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc142b6ec 59b5e749-89f6-4f5d-a177-900ea8fb18a7 /tidb/cdc/default/default/upstream/7489360433438143520 {"id":7489360433438143520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2742.out cli changefeed create --start-ts=457114287038267393 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-fast-fail + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:45 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/323d736e-cd3e-4acc-8b2a-2fa517b5ad93 {"id":"323d736e-cd3e-4acc-8b2a-2fa517b5ad93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752623} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13e3be9 323d736e-cd3e-4acc-8b2a-2fa517b5ad93 /tidb/cdc/default/default/upstream/7489360435876110807 {"id":7489360435876110807,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/323d736e-cd3e-4acc-8b2a-2fa517b5ad93 {"id":"323d736e-cd3e-4acc-8b2a-2fa517b5ad93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752623} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13e3be9 323d736e-cd3e-4acc-8b2a-2fa517b5ad93 /tidb/cdc/default/default/upstream/7489360435876110807 {"id":7489360435876110807,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/323d736e-cd3e-4acc-8b2a-2fa517b5ad93 {"id":"323d736e-cd3e-4acc-8b2a-2fa517b5ad93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752623} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13e3be9 323d736e-cd3e-4acc-8b2a-2fa517b5ad93 /tidb/cdc/default/default/upstream/7489360435876110807 {"id":7489360435876110807,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2727.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ Create changefeed successfully! ID: changefeed-fast-fail Info: {"upstream_id":7489360433438143520,"namespace":"default","id":"changefeed-fast-fail","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:45.869475401+08:00","start_ts":457114287038267393,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114287038267393,"checkpoint_ts":457114287038267393,"checkpoint_time":"2025-04-04 15:43:40.843"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:45 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8cb51d5a-d756-4568-a983-c6ae1eae1bd4 {"id":"8cb51d5a-d756-4568-a983-c6ae1eae1bd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13fd6e9 8cb51d5a-d756-4568-a983-c6ae1eae1bd4 /tidb/cdc/default/default/upstream/7489360437276853351 {"id":7489360437276853351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8cb51d5a-d756-4568-a983-c6ae1eae1bd4 {"id":"8cb51d5a-d756-4568-a983-c6ae1eae1bd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13fd6e9 8cb51d5a-d756-4568-a983-c6ae1eae1bd4 /tidb/cdc/default/default/upstream/7489360437276853351 {"id":7489360437276853351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8cb51d5a-d756-4568-a983-c6ae1eae1bd4 {"id":"8cb51d5a-d756-4568-a983-c6ae1eae1bd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13fd6e9 8cb51d5a-d756-4568-a983-c6ae1eae1bd4 /tidb/cdc/default/default/upstream/7489360437276853351 {"id":7489360437276853351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 350906b5-0f3d-4853-9a63-5a93ef1a1009 Info: {"upstream_id":7489360437276853351,"namespace":"default","id":"350906b5-0f3d-4853-9a63-5a93ef1a1009","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:45.80112564+08:00","start_ts":457114287107211265,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114287107211265,"checkpoint_ts":457114287107211265,"checkpoint_time":"2025-04-04 15:43:41.106"} table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2711.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 4 15:43:45 CST 2025] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.56305632.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 990514b1-a798-4ac2-a6c0-bcf636f3bb95 Info: {"upstream_id":7489360435876110807,"namespace":"default","id":"990514b1-a798-4ac2-a6c0-bcf636f3bb95","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:43:46.21247544+08:00","start_ts":457114288415309827,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288415309827,"checkpoint_ts":457114288415309827,"checkpoint_time":"2025-04-04 15:43:46.096"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:46 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3295 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea3d583-a1ad-4318-bceb-08f661d6d294 {"id":"eea3d583-a1ad-4318-bceb-08f661d6d294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e952de eea3d583-a1ad-4318-bceb-08f661d6d294 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360347588497912,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:25.571065073+08:00","start-ts":457114281707307009,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:43:44.413476771+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114287972548610} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114284407390210,"min-table-barrier-ts":457114284407390210,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea3d583-a1ad-4318-bceb-08f661d6d294 {"id":"eea3d583-a1ad-4318-bceb-08f661d6d294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e952de eea3d583-a1ad-4318-bceb-08f661d6d294 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360347588497912,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:25.571065073+08:00","start-ts":457114281707307009,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:43:44.413476771+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114287972548610} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114284407390210,"min-table-barrier-ts":457114284407390210,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea3d583-a1ad-4318-bceb-08f661d6d294 {"id":"eea3d583-a1ad-4318-bceb-08f661d6d294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e952de eea3d583-a1ad-4318-bceb-08f661d6d294 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360347588497912,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:25.571065073+08:00","start-ts":457114281707307009,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:43:44.413476771+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114287972548610} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114284407390210,"min-table-barrier-ts":457114284407390210,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + set +x + tso='457114288242556929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114288242556929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:46 CST 2025] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.27972799.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:46 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15c367de-5976-4f3b-8014-3a104dbcee0b {"id":"15c367de-5976-4f3b-8014-3a104dbcee0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1492ed4 15c367de-5976-4f3b-8014-3a104dbcee0b /tidb/cdc/default/default/upstream/7489360450719664448 {"id":7489360450719664448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15c367de-5976-4f3b-8014-3a104dbcee0b {"id":"15c367de-5976-4f3b-8014-3a104dbcee0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1492ed4 15c367de-5976-4f3b-8014-3a104dbcee0b /tidb/cdc/default/default/upstream/7489360450719664448 {"id":7489360450719664448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15c367de-5976-4f3b-8014-3a104dbcee0b {"id":"15c367de-5976-4f3b-8014-3a104dbcee0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752624} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1492ed4 15c367de-5976-4f3b-8014-3a104dbcee0b /tidb/cdc/default/default/upstream/7489360450719664448 {"id":7489360450719664448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114284407390210, "checkpoint_time": "2025-04-04 15:43:30.807", "error": { "time": "2025-04-04T15:43:44.413476771+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114284407390210, "checkpoint_time": "2025-04-04 15:43:30.807", "error": { "time": "2025-04-04T15:43:44.413476771+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114284407390210, "checkpoint_time": "2025-04-04 15:43:30.807", "error": { "time": "2025-04-04T15:43:44.413476771+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114284407390210, '"checkpoint_time":' '"2025-04-04' '15:43:30.807",' '"error":' '{' '"time":' '"2025-04-04T15:43:44.413476771+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114284407390210, '"checkpoint_time":' '"2025-04-04' '15:43:30.807",' '"error":' '{' '"time":' '"2025-04-04T15:43:44.413476771+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5802.out cli changefeed remove -c changefeed-error table ddl_manager.finish_mark not exists for 22-th check, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' run task failed 1-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-fast-fail failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-fast-fail + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-fast-fail -s table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later table new_ci_collation_test.t1 exists table new_ci_collation_test.t2 not exists for 1-th check, retry later table partition_table.finish_mark not exists for 5-th check, retry later + info='{ "upstream_id": 7489360433438143520, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114287038267393, "checkpoint_time": "2025-04-04 15:43:40.843", "error": { "time": "2025-04-04T15:43:45.947423824+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7489360433438143520, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114287038267393, "checkpoint_time": "2025-04-04 15:43:40.843", "error": { "time": "2025-04-04T15:43:45.947423824+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7489360433438143520, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114287038267393, "checkpoint_time": "2025-04-04 15:43:40.843", "error": { "time": "2025-04-04T15:43:45.947423824+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489360433438143520, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 457114287038267393, '"checkpoint_time":' '"2025-04-04' '15:43:40.843",' '"error":' '{' '"time":' '"2025-04-04T15:43:45.947423824+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360433438143520, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 457114287038267393, '"checkpoint_time":' '"2025-04-04' '15:43:40.843",' '"error":' '{' '"time":' '"2025-04-04T15:43:45.947423824+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully Changefeed remove successfully. ID: changefeed-error CheckpointTs: 457114284407390210 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS + set +x [Fri Apr 4 15:43:47 CST 2025] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.27632765.out server --log-file /tmp/tidb_cdc_test/bank/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8400 failed: Connection refused * Failed to connect to 127.0.0.1 port 8400: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114288496836609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114288496836609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:47 CST 2025] <<<<<< START cdc server in many_pk_or_uk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.27422744.out server --log-file /tmp/tidb_cdc_test/many_pk_or_uk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/many_pk_or_uk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:48 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2767.out cli changefeed create --start-ts=457114287679733762 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table clustered_index_test.t0 exists table clustered_index_test.t1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Create changefeed successfully! ID: e0d85ba6-6e52-4985-85ed-88baeff711a3 Info: {"upstream_id":7489360440316725357,"namespace":"default","id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:48.670116938+08:00","start_ts":457114287679733762,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114287679733762,"checkpoint_ts":457114287679733762,"checkpoint_time":"2025-04-04 15:43:43.290"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:48 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2dbd90ad-c896-419b-8fbf-2b658c27d9c1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {"id":"2dbd90ad-c896-419b-8fbf-2b658c27d9c1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752626} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb4dc 2dbd90ad-c896-419b-8fbf-2b658c27d9c1 /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2dbd90ad-c896-419b-8fbf-2b658c27d9c1 {"id":"2dbd90ad-c896-419b-8fbf-2b658c27d9c1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752626} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb4dc 2dbd90ad-c896-419b-8fbf-2b658c27d9c1 /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2dbd90ad-c896-419b-8fbf-2b658c27d9c1 {"id":"2dbd90ad-c896-419b-8fbf-2b658c27d9c1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752626} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb4dc 2dbd90ad-c896-419b-8fbf-2b658c27d9c1 /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 2-th time... table consistent_replicate_gbk.GBKTABLE not exists for 1-th check, retry later + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully table partition_table.finish_mark not exists for 6-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:49 CST 2025] <<<<<< run test case changefeed_fast_fail success! >>>>>> table event_filter.t_alter exists table event_filter.t_name not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 23-th check, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table owner_resign.t1 exists check diff failed 1-th time, retry later table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later table new_ci_collation_test.t2 exists table new_ci_collation_test.t3 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea0a80d2-f30e-4212-aea6-dfb0314d030c {"id":"ea0a80d2-f30e-4212-aea6-dfb0314d030c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752627} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14a81eb ea0a80d2-f30e-4212-aea6-dfb0314d030c /tidb/cdc/default/default/upstream/7489360441792358903 {"id":7489360441792358903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea0a80d2-f30e-4212-aea6-dfb0314d030c {"id":"ea0a80d2-f30e-4212-aea6-dfb0314d030c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752627} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14a81eb ea0a80d2-f30e-4212-aea6-dfb0314d030c /tidb/cdc/default/default/upstream/7489360441792358903 {"id":7489360441792358903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea0a80d2-f30e-4212-aea6-dfb0314d030c {"id":"ea0a80d2-f30e-4212-aea6-dfb0314d030c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752627} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14a81eb ea0a80d2-f30e-4212-aea6-dfb0314d030c /tidb/cdc/default/default/upstream/7489360441792358903 {"id":7489360441792358903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2856.out cli changefeed create --start-ts=457114288242556929 --sink-uri=mysql://normal:123456@127.0.0.1:3306/tidb-txn-mode=pessimistic + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' Create changefeed successfully! ID: 5166e1a2-48ac-44ca-b1be-acc6d89eff44 Info: {"upstream_id":7489360441792358903,"namespace":"default","id":"5166e1a2-48ac-44ca-b1be-acc6d89eff44","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/tidb-txn-mode=pessimistic","create_time":"2025-04-04T15:43:50.351890492+08:00","start_ts":457114288242556929,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288242556929,"checkpoint_ts":457114288242556929,"checkpoint_time":"2025-04-04 15:43:45.437"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... table clustered_index_test.t1 exists table clustered_index_test.t2 not exists for 1-th check, retry later check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:43:50 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.59005902.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8400 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17ee557c-13be-4b20-b4c6-3d76769f70df {"id":"17ee557c-13be-4b20-b4c6-3d76769f70df","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffc13e6b12 17ee557c-13be-4b20-b4c6-3d76769f70df /tidb/cdc/default/default/upstream/7489360444094380970 {"id":7489360444094380970,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17ee557c-13be-4b20-b4c6-3d76769f70df {"id":"17ee557c-13be-4b20-b4c6-3d76769f70df","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffc13e6b12 17ee557c-13be-4b20-b4c6-3d76769f70df /tidb/cdc/default/default/upstream/7489360444094380970 {"id":7489360444094380970,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17ee557c-13be-4b20-b4c6-3d76769f70df {"id":"17ee557c-13be-4b20-b4c6-3d76769f70df","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffc13e6b12 17ee557c-13be-4b20-b4c6-3d76769f70df /tidb/cdc/default/default/upstream/7489360444094380970 {"id":7489360444094380970,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2829.out cli changefeed create --sink-uri=blackhole:// -c changefeed-for-find-finished-ts --server http://127.0.0.1:8400 table consistent_replicate_gbk.GBKTABLE exists table consistent_replicate_gbk.check1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b00b12b4-1732-4b14-8b78-56c85bb28f9f {"id":"b00b12b4-1732-4b14-8b78-56c85bb28f9f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14eafe7 b00b12b4-1732-4b14-8b78-56c85bb28f9f /tidb/cdc/default/default/upstream/7489360450106133187 {"id":7489360450106133187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b00b12b4-1732-4b14-8b78-56c85bb28f9f {"id":"b00b12b4-1732-4b14-8b78-56c85bb28f9f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14eafe7 b00b12b4-1732-4b14-8b78-56c85bb28f9f /tidb/cdc/default/default/upstream/7489360450106133187 {"id":7489360450106133187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b00b12b4-1732-4b14-8b78-56c85bb28f9f {"id":"b00b12b4-1732-4b14-8b78-56c85bb28f9f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14eafe7 b00b12b4-1732-4b14-8b78-56c85bb28f9f /tidb/cdc/default/default/upstream/7489360450106133187 {"id":7489360450106133187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2797.out cli changefeed create --start-ts=457114288496836609 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: changefeed-for-find-finished-ts Info: {"upstream_id":7489360444094380970,"namespace":"default","id":"changefeed-for-find-finished-ts","sink_uri":"blackhole:","create_time":"2025-04-04T15:43:51.164586135+08:00","start_ts":457114289712922631,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114289712922631,"checkpoint_ts":457114289712922631,"checkpoint_time":"2025-04-04 15:43:51.046"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table partition_table.finish_mark not exists for 7-th check, retry later Create changefeed successfully! ID: 5247659b-ca78-4a40-a346-05e72a2183fc Info: {"upstream_id":7489360450106133187,"namespace":"default","id":"5247659b-ca78-4a40-a346-05e72a2183fc","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:51.310196772+08:00","start_ts":457114288496836609,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288496836609,"checkpoint_ts":457114288496836609,"checkpoint_time":"2025-04-04 15:43:46.407"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table event_filter.t_name exists table event_filter.t_name1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 24-th check, retry later check diff successfully check diff successfully + set +x go: downloading go1.23.4 (linux/amd64) check diff failed 5-th time, retry later table new_ci_collation_test.t3 exists table new_ci_collation_test.t4 not exists for 1-th check, retry later [Fri Apr 4 15:43:51 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.29112913.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "is-owner": true, run task successfully wait process cdc.test exit for 1-th time... table clustered_index_test.t2 exists check table exists success check diff failed 1-th time, retry later + set +x + GO111MODULE=on + go run bank.go case.go -u 'root@tcp(127.0.0.1:4000)/bank' -d 'root@tcp(127.0.0.1:3306)/bank' --test-round=20000 -a 127.0.0.1:20080 go: downloading go1.23.4 (linux/amd64) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... + set +x go: downloading go1.23.4 (linux/amd64) check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:53 CST 2025] <<<<<< run test case tiflash success! >>>>>> owner pid: 2710 owner id f689ec2e-4613-4696-ad41-cd570d6b16cc [Fri Apr 4 15:43:53 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.28632865.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 25-th check, retry later table consistent_replicate_gbk.check1 not exists for 2-th check, retry later table new_ci_collation_test.t4 exists table new_ci_collation_test.t5 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table partition_table.finish_mark not exists for 8-th check, retry later table event_filter.t_name1 exists table event_filter.t_name2 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:53 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/93996fca-fc03-4156-aa25-5f635bc8b65a {"id":"93996fca-fc03-4156-aa25-5f635bc8b65a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9533f 93996fca-fc03-4156-aa25-5f635bc8b65a /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/93996fca-fc03-4156-aa25-5f635bc8b65a {"id":"93996fca-fc03-4156-aa25-5f635bc8b65a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9533f 93996fca-fc03-4156-aa25-5f635bc8b65a /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/93996fca-fc03-4156-aa25-5f635bc8b65a {"id":"93996fca-fc03-4156-aa25-5f635bc8b65a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9533f 93996fca-fc03-4156-aa25-5f635bc8b65a /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5951.out cli changefeed create --start-ts=457114281707307009 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-1 check diff failed 6-th time, retry later Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7489360347588497912,"namespace":"default","id":"changefeed-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:54.341499636+08:00","start_ts":457114281707307009,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009,"checkpoint_time":"2025-04-04 15:43:20.507"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:54 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/d521d148-2502-4e5d-8467-fd3f88dea099 {UpstreamID:7489360444900490223 Namespace:default ID:d521d148-2502-4e5d-8467-fd3f88dea099 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:45.620143427 +0800 CST StartTs:457114288267984898 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c1d8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288281092109} {CheckpointTs:457114290378244108 MinTableBarrierTs:457114290640650249 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/capture/e05a6ea4-9855-4477-92bc-332b64c67b48 {"id":"e05a6ea4-9855-4477-92bc-332b64c67b48","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752632} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/__cdc_meta__/owner/223195ffc1408973 e05a6ea4-9855-4477-92bc-332b64c67b48 /tidb/cdc/default/default/changefeed/info/d521d148-2502-4e5d-8467-fd3f88dea099 {"upstream-id":7489360444900490223,"namespace":"default","changefeed-id":"d521d148-2502-4e5d-8467-fd3f88dea099","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:45.620143427+08:00","start-ts":457114288267984898,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288281092109} /tidb/cdc/default/default/changefeed/status/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":457114290378244108,"min-table-barrier-ts":457114290640650249,"admin-job-type":0} /tidb/cdc/default/default/task/position/41975e39-928b-4bf0-88c5-dffa10baf9fa/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e05a6ea4-9855-4477-92bc-332b64c67b48/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/d521d148-2502-4e5d-8467-fd3f88dea099 {UpstreamID:7489360444900490223 Namespace:default ID:d521d148-2502-4e5d-8467-fd3f88dea099 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:45.620143427 +0800 CST StartTs:457114288267984898 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c1d8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288281092109} {CheckpointTs:457114290378244108 MinTableBarrierTs:457114290640650249 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/capture/e05a6ea4-9855-4477-92bc-332b64c67b48 {"id":"e05a6ea4-9855-4477-92bc-332b64c67b48","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752632} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/__cdc_meta__/owner/223195ffc1408973 e05a6ea4-9855-4477-92bc-332b64c67b48 /tidb/cdc/default/default/changefeed/info/d521d148-2502-4e5d-8467-fd3f88dea099 {"upstream-id":7489360444900490223,"namespace":"default","changefeed-id":"d521d148-2502-4e5d-8467-fd3f88dea099","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:45.620143427+08:00","start-ts":457114288267984898,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288281092109} /tidb/cdc/default/default/changefeed/status/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":457114290378244108,"min-table-barrier-ts":457114290640650249,"admin-job-type":0} /tidb/cdc/default/default/task/position/41975e39-928b-4bf0-88c5-dffa10baf9fa/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e05a6ea4-9855-4477-92bc-332b64c67b48/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/d521d148-2502-4e5d-8467-fd3f88dea099 {UpstreamID:7489360444900490223 Namespace:default ID:d521d148-2502-4e5d-8467-fd3f88dea099 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:45.620143427 +0800 CST StartTs:457114288267984898 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c1d8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288281092109} {CheckpointTs:457114290378244108 MinTableBarrierTs:457114290640650249 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41975e39-928b-4bf0-88c5-dffa10baf9fa {"id":"41975e39-928b-4bf0-88c5-dffa10baf9fa","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/capture/e05a6ea4-9855-4477-92bc-332b64c67b48 {"id":"e05a6ea4-9855-4477-92bc-332b64c67b48","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752632} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14088d6 41975e39-928b-4bf0-88c5-dffa10baf9fa /tidb/cdc/default/__cdc_meta__/owner/223195ffc1408973 e05a6ea4-9855-4477-92bc-332b64c67b48 /tidb/cdc/default/default/changefeed/info/d521d148-2502-4e5d-8467-fd3f88dea099 {"upstream-id":7489360444900490223,"namespace":"default","changefeed-id":"d521d148-2502-4e5d-8467-fd3f88dea099","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:45.620143427+08:00","start-ts":457114288267984898,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288281092109} /tidb/cdc/default/default/changefeed/status/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":457114290378244108,"min-table-barrier-ts":457114290640650249,"admin-job-type":0} /tidb/cdc/default/default/task/position/41975e39-928b-4bf0-88c5-dffa10baf9fa/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e05a6ea4-9855-4477-92bc-332b64c67b48/d521d148-2502-4e5d-8467-fd3f88dea099 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360444900490223 {"id":7489360444900490223,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 26-th check, retry later table consistent_replicate_gbk.check1 exists check diff successfully table new_ci_collation_test.t5 exists check diff failed 1-th time, retry later table partition_table.finish_mark exists check diff successfully + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed wait process cdc.test exit for 1-th time... table event_filter.t_name2 exists table event_filter.t_name3 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1000 0 --:--:-- --:--:-- --:--:-- 1000 + info='{"state":"normal","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009}' + echo '{"state":"normal","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009}' {"state":"normal","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009} ++ echo '{"state":"normal","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009}' ++ jq -r .state + state=normal + [[ ! normal == \w\a\r\n\i\n\g ]] + echo 'changefeed state normal does not equal to warning' changefeed state normal does not equal to warning + exit 1 run task failed 1-th time, retry later check diff successfully Verifying downstream PD is started... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:56 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017f9c20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114288151592974 MinTableBarrierTs:457114290890997775 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/df027e6f-83c2-4969-9149-9af5536214ad {"id":"df027e6f-83c2-4969-9149-9af5536214ad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460659 df027e6f-83c2-4969-9149-9af5536214ad /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114288151592974,"min-table-barrier-ts":457114290890997775,"admin-job-type":0} /tidb/cdc/default/default/task/position/df027e6f-83c2-4969-9149-9af5536214ad/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f689ec2e-4613-4696-ad41-cd570d6b16cc/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017f9c20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114288151592974 MinTableBarrierTs:457114290890997775 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/df027e6f-83c2-4969-9149-9af5536214ad {"id":"df027e6f-83c2-4969-9149-9af5536214ad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460659 df027e6f-83c2-4969-9149-9af5536214ad /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114288151592974,"min-table-barrier-ts":457114290890997775,"admin-job-type":0} /tidb/cdc/default/default/task/position/df027e6f-83c2-4969-9149-9af5536214ad/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f689ec2e-4613-4696-ad41-cd570d6b16cc/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017f9c20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114288151592974 MinTableBarrierTs:457114290890997775 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/df027e6f-83c2-4969-9149-9af5536214ad {"id":"df027e6f-83c2-4969-9149-9af5536214ad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/f689ec2e-4613-4696-ad41-cd570d6b16cc {"id":"f689ec2e-4613-4696-ad41-cd570d6b16cc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14605f0 f689ec2e-4613-4696-ad41-cd570d6b16cc /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460659 df027e6f-83c2-4969-9149-9af5536214ad /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114288151592974,"min-table-barrier-ts":457114290890997775,"admin-job-type":0} /tidb/cdc/default/default/task/position/df027e6f-83c2-4969-9149-9af5536214ad/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f689ec2e-4613-4696-ad41-cd570d6b16cc/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "f689ec2e-4613-4696-ad41-cd570d6b16cc" | grep -v cluster_id | grep id check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:56 CST 2025] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.58125814.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:56 CST 2025] <<<<<< run test case partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/sys v0.28.0 table ddl_manager.finish_mark not exists for 27-th check, retry later go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/net v0.33.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading golang.org/x/sync v0.10.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/multierr v1.11.0 go: downloading google.golang.org/protobuf v1.35.2 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading golang.org/x/sys v0.28.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda table event_filter.t_name3 exists check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 3162 0 --:--:-- --:--:-- --:--:-- 3162 + info='{"state":"warning","resolved_ts":457114282087415809,"checkpoint_ts":457114282087415809,"last_warning":{"time":"2025-04-04T15:43:56.020643025+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":457114282087415809,"checkpoint_ts":457114282087415809,"last_warning":{"time":"2025-04-04T15:43:56.020643025+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":457114282087415809,"checkpoint_ts":457114282087415809,"last_warning":{"time":"2025-04-04T15:43:56.020643025+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":457114282087415809,"checkpoint_ts":457114282087415809,"last_warning":{"time":"2025-04-04T15:43:56.020643025+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":457114282087415809,"checkpoint_ts":457114282087415809,"last_warning":{"time":"2025-04-04T15:43:56.020643025+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6016.out cli changefeed remove -c changefeed-error-1 "cluster-id": "default" "id": "df027e6f-83c2-4969-9149-9af5536214ad", "cluster-id": "default" run task successfully check diff failed 2-th time, retry later go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 table event_filter.finish_mark exists check diff failed 1-th time, retry later check diff failed 4-th time, retry later check diff failed 1-th time, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 457114282087415809 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table ddl_manager.finish_mark not exists for 28-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) capture_id: df027e6f-83c2-4969-9149-9af5536214ad go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_update_to_no_batch/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 check diff failed 3-th time, retry later wait process 2710 exit for 1-th time... go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/stretchr/testify v1.10.0 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/tidwall/btree v1.7.0 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/642e72b5-259b-4b9b-bc9d-967fa66fc98f {UpstreamID:7489360452397051178 Namespace:default ID:642e72b5-259b-4b9b-bc9d-967fa66fc98f SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.905453588 +0800 CST StartTs:457114289137254401 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003954090 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289150623746} {CheckpointTs:457114289189683224 MinTableBarrierTs:457114291863814151 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105ee243-b370-4d01-b634-e75db2cadf5a {"id":"105ee243-b370-4d01-b634-e75db2cadf5a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb586 105ee243-b370-4d01-b634-e75db2cadf5a /tidb/cdc/default/default/changefeed/info/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"upstream-id":7489360452397051178,"namespace":"default","changefeed-id":"642e72b5-259b-4b9b-bc9d-967fa66fc98f","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.905453588+08:00","start-ts":457114289137254401,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289150623746} /tidb/cdc/default/default/changefeed/status/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":457114289189683224,"min-table-barrier-ts":457114291863814151,"admin-job-type":0} /tidb/cdc/default/default/task/position/105ee243-b370-4d01-b634-e75db2cadf5a/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/642e72b5-259b-4b9b-bc9d-967fa66fc98f {UpstreamID:7489360452397051178 Namespace:default ID:642e72b5-259b-4b9b-bc9d-967fa66fc98f SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.905453588 +0800 CST StartTs:457114289137254401 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003954090 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289150623746} {CheckpointTs:457114289189683224 MinTableBarrierTs:457114291863814151 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105ee243-b370-4d01-b634-e75db2cadf5a {"id":"105ee243-b370-4d01-b634-e75db2cadf5a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb586 105ee243-b370-4d01-b634-e75db2cadf5a /tidb/cdc/default/default/changefeed/info/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"upstream-id":7489360452397051178,"namespace":"default","changefeed-id":"642e72b5-259b-4b9b-bc9d-967fa66fc98f","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.905453588+08:00","start-ts":457114289137254401,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289150623746} /tidb/cdc/default/default/changefeed/status/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":457114289189683224,"min-table-barrier-ts":457114291863814151,"admin-job-type":0} /tidb/cdc/default/default/task/position/105ee243-b370-4d01-b634-e75db2cadf5a/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/642e72b5-259b-4b9b-bc9d-967fa66fc98f {UpstreamID:7489360452397051178 Namespace:default ID:642e72b5-259b-4b9b-bc9d-967fa66fc98f SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.905453588 +0800 CST StartTs:457114289137254401 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003954090 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289150623746} {CheckpointTs:457114289189683224 MinTableBarrierTs:457114291863814151 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114289189683224, checkpointTs: 457114289189683224, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105ee243-b370-4d01-b634-e75db2cadf5a {"id":"105ee243-b370-4d01-b634-e75db2cadf5a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14cb586 105ee243-b370-4d01-b634-e75db2cadf5a /tidb/cdc/default/default/changefeed/info/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"upstream-id":7489360452397051178,"namespace":"default","changefeed-id":"642e72b5-259b-4b9b-bc9d-967fa66fc98f","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.905453588+08:00","start-ts":457114289137254401,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289150623746} /tidb/cdc/default/default/changefeed/status/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":457114289189683224,"min-table-barrier-ts":457114291863814151,"admin-job-type":0} /tidb/cdc/default/default/task/position/105ee243-b370-4d01-b634-e75db2cadf5a/642e72b5-259b-4b9b-bc9d-967fa66fc98f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360452397051178 {"id":7489360452397051178,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.29.11 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading cloud.google.com/go/iam v1.1.7 wait process 2710 exit for 2-th time... + set +x go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading golang.org/x/text v0.21.0 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 check diff failed 1-th time, retry later check diff failed 2-th time, retry later go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2710) - No such process wait process 2710 exit for 3-th time... process 2710 already exit cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep df027e6f-83c2-4969-9149-9af5536214ad -A1 | grep '"is-owner": true' go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda check diff failed 5-th time, retry later go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 wait process cdc.test exit for 2-th time... go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 table ddl_manager.finish_mark not exists for 29-th check, retry later go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d check diff failed 4-th time, retry later + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:44:01 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60726074.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "recordcount"="10" "mysql.user"="root" "mysql.port"="4000" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.db"="testSync" "threadcount"="2" "operationcount"="0" "readproportion"="0" "readallfields"="true" "scanproportion"="0" "dotransactions"="false" "updateproportion"="0" "insertproportion"="0" ********************************************** Run finished, takes 4.820252ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3379.3, Avg(us): 891, Min(us): 531, Max(us): 1879, 95th(us): 2000, 99th(us): 2000 go: downloading k8s.io/apimachinery v0.29.11 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b check diff failed 2-th time, retry later go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 "is-owner": true, run task successfully test_kill_owner: pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/batch_update_to_no_batch Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 6-th time, retry later check diff successfully go: downloading github.com/pierrec/lz4 v2.6.1+incompatible =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 30-th check, retry later wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360536582657844 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06f7c00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:9509, start at 2025-04-04 15:44:02.05846602 +0800 CST m=+2.965578862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:02.071 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:02.082 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:02.082 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360536582657844 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06f7c00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:9509, start at 2025-04-04 15:44:02.05846602 +0800 CST m=+2.965578862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:02.071 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:02.082 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:02.082 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360541639913849 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06f7a8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:9566, start at 2025-04-04 15:44:02.074043067 +0800 CST m=+2.899085646 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:02.084 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:02.075 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:02.075 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_owner [Fri Apr 4 15:44:03 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.30643066.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 3-th time, retry later wait process cdc.test exit for 2-th time... go: downloading github.com/pierrec/lz4 v2.6.1+incompatible Verifying downstream PD is started... check diff failed 5-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:04 CST 2025] <<<<<< run test case event_filter success! >>>>>> check diff failed 7-th time, retry later go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:04 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca {"id":"3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752642} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e953cc 3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca {"id":"3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752642} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e953cc 3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca {"id":"3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752642} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e953cc 3d508cc9-c24a-4ce7-bc4a-c1bdc29fdcca /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6124.out cli changefeed create --start-ts=457114281707307009 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/jmespath/go-jmespath v0.4.0 check_redo_resolved_ts 642e72b5-259b-4b9b-bc9d-967fa66fc98f 457114293423570948 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/642e72b5-259b-4b9b-bc9d-967fa66fc98f/meta global resolved ts 457114292060422152 not forward to 457114293423570948 run task failed 1-th time, retry later Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7489360347588497912,"namespace":"default","id":"changefeed-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:05.234663281+08:00","start_ts":457114281707307009,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114281707307009,"checkpoint_ts":457114281707307009,"checkpoint_time":"2025-04-04 15:43:20.507"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 31-th check, retry later go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.10860.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 4-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_with_exists/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:06 CST 2025] <<<<<< run test case ddl_with_exists success! >>>>>> + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:44:05.314047651+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:44:05.314047651+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114281707307009, "checkpoint_time": "2025-04-04 15:43:20.507", "error": { "time": "2025-04-04T15:44:05.314047651+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114281707307009, '"checkpoint_time":' '"2025-04-04' '15:43:20.507",' '"error":' '{' '"time":' '"2025-04-04T15:44:05.314047651+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114281707307009, '"checkpoint_time":' '"2025-04-04' '15:43:20.507",' '"error":' '{' '"time":' '"2025-04-04T15:44:05.314047651+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114281707307009 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6191.out cli changefeed remove -c changefeed-error-2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360451812755993 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0722b0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:1305, start at 2025-04-04 15:44:04.815971172 +0800 CST m=+24.209818977 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:04.823 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:04.780 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:04.780 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360451812755993 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0722b0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:1305, start at 2025-04-04 15:44:04.815971172 +0800 CST m=+24.209818977 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:04.823 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:04.780 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:04.780 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360464420607363 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05c578000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:1384, start at 2025-04-04 15:43:42.443223782 +0800 CST m=+1.762702723 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:42.450 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:42.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:42.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:06 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039a3cb0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114291677429774 MinTableBarrierTs:457114293787688965 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114291677429774,"min-table-barrier-ts":457114293787688965,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039a3cb0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114291677429774 MinTableBarrierTs:457114293787688965 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114291677429774,"min-table-barrier-ts":457114293787688965,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039a3cb0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114291677429774 MinTableBarrierTs:457114293787688965 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291677429774, checkpointTs: 457114291677429774, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114291677429774,"min-table-barrier-ts":457114293787688965,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table ddl_manager.finish_mark not exists for 32-th check, retry later check diff failed 8-th time, retry later + set +x + tso='457114293588983809 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114293588983809 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114293588983809 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:44:07 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1089010892.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 457114281707307009 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check_redo_resolved_ts 642e72b5-259b-4b9b-bc9d-967fa66fc98f 457114293423570948 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/642e72b5-259b-4b9b-bc9d-967fa66fc98f/meta global resolved ts 457114292060422152 not forward to 457114293423570948 run task failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later "is-owner": true, run task successfully check diff successfully + set +x wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 33-th check, retry later check diff failed 9-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.cli.2831.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release owner pid: 3069 owner id 31bdf774-7014-477a-a680-b07428575b9c [Fri Apr 4 15:44:09 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.31733175.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:09.895 +08:00] [INFO] [case.go:450] ["open db success"] [dsn=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:09.896 +08:00] [INFO] [case.go:450] ["open db success"] [dsn=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:09.896 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq0] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:10 CST 2025] <<<<<< run test case new_ci_collation success! >>>>>> wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:10 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.62346236.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:10.423 +08:00] [INFO] [case.go:370] ["insert 100 takes 8.1367ms"] [query="INSERT IGNORE INTO accounts_seq0 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:44:10.424 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq1] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:10 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a3501407-66f5-421d-8d15-ba56ce7f1ac1 {"id":"a3501407-66f5-421d-8d15-ba56ce7f1ac1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc19c52e8 a3501407-66f5-421d-8d15-ba56ce7f1ac1 /tidb/cdc/default/default/upstream/7489360536582657844 {"id":7489360536582657844,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a3501407-66f5-421d-8d15-ba56ce7f1ac1 {"id":"a3501407-66f5-421d-8d15-ba56ce7f1ac1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc19c52e8 a3501407-66f5-421d-8d15-ba56ce7f1ac1 /tidb/cdc/default/default/upstream/7489360536582657844 {"id":7489360536582657844,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a3501407-66f5-421d-8d15-ba56ce7f1ac1 {"id":"a3501407-66f5-421d-8d15-ba56ce7f1ac1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc19c52e8 a3501407-66f5-421d-8d15-ba56ce7f1ac1 /tidb/cdc/default/default/upstream/7489360536582657844 {"id":7489360536582657844,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114293588983809 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.10947.out cli changefeed create --start-ts=457114293588983809 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360536582657844,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:10.796101139+08:00","start_ts":457114293588983809,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114293588983809,"checkpoint_ts":457114293588983809,"checkpoint_time":"2025-04-04 15:44:05.832"} PASS check diff successfully + set +x + tso='457114294532177921 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114294532177921 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:10 CST 2025] <<<<<< START cdc server in tidb_mysql_test case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.28692871.out server --log-file /tmp/tidb_cdc_test/tidb_mysql_test/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tidb_mysql_test/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 coverage: 2.5% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m [2025/04/04 15:44:10.939 +08:00] [INFO] [case.go:370] ["insert 100 takes 8.065337ms"] [query="INSERT IGNORE INTO accounts_seq1 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:44:10.940 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq2] [Pipeline] } check diff failed 10-th time, retry later Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2025/04/04 15:44:11.453 +08:00] [INFO] [case.go:370] ["insert 100 takes 5.271929ms"] [query="INSERT IGNORE INTO accounts_seq2 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:44:11.453 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq3] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [Pipeline] // stage table ddl_manager.finish_mark not exists for 34-th check, retry later [Pipeline] } [2025/04/04 15:44:11.965 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.8315ms"] [query="INSERT IGNORE INTO accounts_seq3 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:44:11.966 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq4] [Pipeline] // container Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } check_redo_resolved_ts 642e72b5-259b-4b9b-bc9d-967fa66fc98f 457114293423570948 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/642e72b5-259b-4b9b-bc9d-967fa66fc98f/meta run task successfully [Pipeline] // withEnv [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // node [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360574444162717 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0774ec000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:3901, start at 2025-04-04 15:44:10.090379169 +0800 CST m=+2.227901729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:10.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:10.092 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:10.092 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360574444162717 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0774ec000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:3901, start at 2025-04-04 15:44:10.090379169 +0800 CST m=+2.227901729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:10.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:10.092 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:10.092 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360580136285106 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0778c40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:3980, start at 2025-04-04 15:44:10.298343104 +0800 CST m=+2.345414040 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:10.332 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:10.340 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:10.340 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 9208 0 --:--:-- --:--:-- --:--:-- 9208 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:44:05.832","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:44:12.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:05.832","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:12.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:05.832","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:12.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:44:05.832' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:05.832","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:12.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:05.832","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:12.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:44:12' + echo 'sink_checkpoint_ts is 2025-04-04' 15:44:05.832 sink_checkpoint_ts is 2025-04-04 15:44:05.832 ++ date -d '2025-04-04 15:44:05.832' +%s + checkpoint_timestamp=1743752645 ++ date -d '2025-04-04 15:44:12' +%s + current_timestamp=1743752652 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' [Pipeline] // podTemplate [2025/04/04 15:44:12.478 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.895278ms"] [query="INSERT IGNORE INTO accounts_seq4 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:44:12.478 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts0] [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } check diff successfully wait process cdc.test exit for 2-th time... + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [2025/04/04 15:44:12.991 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.732752ms"] [query="INSERT IGNORE INTO accounts0 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:44:12.991 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts1] wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003968360 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114295360552967 MinTableBarrierTs:457114295360552967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/capture/47e45ed4-40f5-414c-a579-1656f15c1bb0 {"id":"47e45ed4-40f5-414c-a579-1656f15c1bb0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/__cdc_meta__/owner/223195ffc146073c 47e45ed4-40f5-414c-a579-1656f15c1bb0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114295360552967,"min-table-barrier-ts":457114295360552967,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/47e45ed4-40f5-414c-a579-1656f15c1bb0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003968360 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114295360552967 MinTableBarrierTs:457114295360552967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/capture/47e45ed4-40f5-414c-a579-1656f15c1bb0 {"id":"47e45ed4-40f5-414c-a579-1656f15c1bb0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/__cdc_meta__/owner/223195ffc146073c 47e45ed4-40f5-414c-a579-1656f15c1bb0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114295360552967,"min-table-barrier-ts":457114295360552967,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/47e45ed4-40f5-414c-a579-1656f15c1bb0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003968360 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114295360552967 MinTableBarrierTs:457114295360552967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31bdf774-7014-477a-a680-b07428575b9c {"id":"31bdf774-7014-477a-a680-b07428575b9c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752644} /tidb/cdc/default/__cdc_meta__/capture/47e45ed4-40f5-414c-a579-1656f15c1bb0 {"id":"47e45ed4-40f5-414c-a579-1656f15c1bb0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14606da 31bdf774-7014-477a-a680-b07428575b9c /tidb/cdc/default/__cdc_meta__/owner/223195ffc146073c 47e45ed4-40f5-414c-a579-1656f15c1bb0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114295360552967,"min-table-barrier-ts":457114295360552967,"admin-job-type":0} /tidb/cdc/default/default/task/position/31bdf774-7014-477a-a680-b07428575b9c/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0+ grep -q 'etcd info' ,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/47e45ed4-40f5-414c-a579-1656f15c1bb0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "31bdf774-7014-477a-a680-b07428575b9c" | grep id [2025/04/04 15:44:13.236 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:44:13.237 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:44:13.237 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:44:13.238 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:44:13.241 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:44:13.241 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114292060422152] [resolvedTs=457114294747398150] [2025/04/04 15:44:13.241 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:44:13.242 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:44:13.242 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:44:13.242 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:44:13.242 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:44:13.245 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.41876ms] [2025/04/04 15:44:13.245 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=4.620169ms] [2025/04/04 15:44:13.247 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:44:13.250 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:44:13.251 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:44:13.251 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:44:13.251 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:44:13.252 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:44:13.257 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:44:13.259 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:44:13.259 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:44:13.259 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114292060422148,\"CommitTs\":457114292060422152,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_gbk\",\"Table\":\"GBKTABLE2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:44:13.260 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114292060422152] [DDL="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [2025/04/04 15:44:13.261 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114292060422148] [ddl="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [namespace=] [changefeed=] [error="Error 1050 (42S01): Table 'consistent_replicate_gbk.GBKTABLE2' already exists"] [2025/04/04 15:44:13.504 +08:00] [INFO] [case.go:370] ["insert 100 takes 5.539402ms"] [query="INSERT IGNORE INTO accounts1 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:44:13.505 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts2] [2025/04/04 15:44:13.462 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114294747398150] [2025/04/04 15:44:13.462 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114294747398150] [2025/04/04 15:44:13.562 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=1] [appliedDDLCount=1] [currentCheckpoint=457114294747398150] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:44:13.562 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] Apply redo log successfully wait process cdc.test exit for 2-th time... check diff failed 11-th time, retry later table ddl_manager.finish_mark not exists for 35-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:13 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/504a82f6-67dc-41d8-bb48-7adae34db04c {"id":"504a82f6-67dc-41d8-bb48-7adae34db04c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9542a 504a82f6-67dc-41d8-bb48-7adae34db04c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/504a82f6-67dc-41d8-bb48-7adae34db04c {"id":"504a82f6-67dc-41d8-bb48-7adae34db04c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9542a 504a82f6-67dc-41d8-bb48-7adae34db04c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/504a82f6-67dc-41d8-bb48-7adae34db04c {"id":"504a82f6-67dc-41d8-bb48-7adae34db04c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752650} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0e9542a 504a82f6-67dc-41d8-bb48-7adae34db04c /tidb/cdc/default/default/upstream/7489360347588497912 {"id":7489360347588497912,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6286.out cli changefeed create --start-ts=0 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-initialize-error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [Fri Apr 4 15:44:13 CST 2025] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7489360347588497912,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:14.076563743+08:00","start_ts":457114295732011010,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114295732011010,"checkpoint_ts":457114295732011010,"checkpoint_time":"2025-04-04 15:44:14.007"} PASS [2025/04/04 15:44:14.017 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.30619ms"] [query="INSERT IGNORE INTO accounts2 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:44:14.017 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts3] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:14 CST 2025] <<<<<< run test case owner_resign success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:13 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd3636b5-5d18-453b-966a-fa71b57f4b4c {"id":"dd3636b5-5d18-453b-966a-fa71b57f4b4c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1521841 dd3636b5-5d18-453b-966a-fa71b57f4b4c /tidb/cdc/default/default/upstream/7489360451812755993 {"id":7489360451812755993,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd3636b5-5d18-453b-966a-fa71b57f4b4c {"id":"dd3636b5-5d18-453b-966a-fa71b57f4b4c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1521841 dd3636b5-5d18-453b-966a-fa71b57f4b4c /tidb/cdc/default/default/upstream/7489360451812755993 {"id":7489360451812755993,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd3636b5-5d18-453b-966a-fa71b57f4b4c {"id":"dd3636b5-5d18-453b-966a-fa71b57f4b4c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1521841 dd3636b5-5d18-453b-966a-fa71b57f4b4c /tidb/cdc/default/default/upstream/7489360451812755993 {"id":7489360451812755993,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 142 0 0 100 142 0 1378 --:--:-- --:--:-- --:--:-- 1378 mysql_test start go: downloading github.com/maxshuang/mysql-tester v0.0.2 coverage: 2.4% of statements in github.com/pingcap/tiflow/... "cluster-id": "default" "id": "47e45ed4-40f5-414c-a579-1656f15c1bb0", "cluster-id": "default" run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... [2025/04/04 15:44:14.531 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.901422ms"] [query="INSERT IGNORE INTO accounts3 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:44:14.531 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts4] table test.t1 exists + sleep 5 check diff failed 1-th time, retry later [2025/04/04 15:44:15.044 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.835592ms"] [query="INSERT IGNORE INTO accounts4 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] check diff failed 12-th time, retry later go: downloading github.com/pingcap/tidb/parser v0.0.0-20211122065349-4d16aabf900b go: downloading github.com/pingcap/errors v0.11.5-0.20211009033009-93128226aaa3 go: downloading github.com/go-sql-driver/mysql v1.6.0 go: downloading github.com/sirupsen/logrus v1.8.1 go: downloading go.uber.org/atomic v1.9.0 go: downloading golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1 go: downloading go.uber.org/zap v1.19.1 go: downloading github.com/pingcap/log v0.0.0-20210906054005-afc726e70354 go: downloading golang.org/x/text v0.3.7 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 go: downloading go.uber.org/multierr v1.7.0 table ddl_manager.finish_mark not exists for 36-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s capture_id: 47e45ed4-40f5-414c-a579-1656f15c1bb0 ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '47e45ed4-40f5-414c-a579-1656f15c1bb0' 47e45ed4-40f5-414c-a579-1656f15c1bb0 run task successfully test_hang_up_owner: pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... [Fri Apr 4 15:44:16 CST 2025] <<<<<< START cdc server in batch_update_to_no_batch case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.52135215.out server --log-file /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114295732011010, "checkpoint_time": "2025-04-04 15:44:14.007", "error": { "time": "2025-04-04T15:44:14.257304382+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114295732011010, "checkpoint_time": "2025-04-04 15:44:14.007", "error": { "time": "2025-04-04T15:44:14.257304382+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114295732011010, "checkpoint_time": "2025-04-04 15:44:14.007", "error": { "time": "2025-04-04T15:44:14.257304382+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114295732011010, '"checkpoint_time":' '"2025-04-04' '15:44:14.007",' '"error":' '{' '"time":' '"2025-04-04T15:44:14.257304382+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state check diff failed 13-th time, retry later table ddl_manager.finish_mark not exists for 37-th check, retry later + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_owner [Fri Apr 4 15:44:17 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.33133315.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_owner.server1 --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:17.551 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360596269419294 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07dd60002e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:3986, start at 2025-04-04 15:44:16.734796457 +0800 CST m=+4.989184924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:16.743 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:16.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:16.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360596269419294 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07dd60002e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:3986, start at 2025-04-04 15:44:16.734796457 +0800 CST m=+4.989184924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:16.743 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:16.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:16.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360596638623544 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07b624000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:4067, start at 2025-04-04 15:44:14.249819818 +0800 CST m=+2.432102905 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:14.257 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:14.217 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:14.217 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 38-th check, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6391.out cli changefeed pause -c changefeed-initialize-error [2025/04/04 15:44:19.558 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72939ee1-e765-469d-b04f-70499a3e6291 {"id":"72939ee1-e765-469d-b04f-70499a3e6291","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c07ded 72939ee1-e765-469d-b04f-70499a3e6291 /tidb/cdc/default/default/upstream/7489360574444162717 {"id":7489360574444162717,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72939ee1-e765-469d-b04f-70499a3e6291 {"id":"72939ee1-e765-469d-b04f-70499a3e6291","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c07ded 72939ee1-e765-469d-b04f-70499a3e6291 /tidb/cdc/default/default/upstream/7489360574444162717 {"id":7489360574444162717,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72939ee1-e765-469d-b04f-70499a3e6291 {"id":"72939ee1-e765-469d-b04f-70499a3e6291","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c07ded 72939ee1-e765-469d-b04f-70499a3e6291 /tidb/cdc/default/default/upstream/7489360574444162717 {"id":7489360574444162717,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5271.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' -c test Create changefeed successfully! ID: test Info: {"upstream_id":7489360574444162717,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2025-04-04T15:44:19.816146175+08:00","start_ts":457114297222561796,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297222561796,"checkpoint_ts":457114297222561796,"checkpoint_time":"2025-04-04 15:44:19.693"} PASS =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff failed 14-th time, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 48600 0 --:--:-- --:--:-- --:--:-- 48600 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:44:18.182","puller_resolved_ts":"2025-04-04 15:44:12.382","last_synced_ts":"2025-04-04 15:44:12.882","now_ts":"2025-04-04 15:44:19.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:18.182","puller_resolved_ts":"2025-04-04' '15:44:12.382","last_synced_ts":"2025-04-04' '15:44:12.882","now_ts":"2025-04-04' '15:44:19.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:18.182","puller_resolved_ts":"2025-04-04' '15:44:12.382","last_synced_ts":"2025-04-04' '15:44:12.882","now_ts":"2025-04-04' '15:44:19.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:44:20 CST 2025] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.53175319.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:20 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c6b8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114296146984967 MinTableBarrierTs:457114297195560966 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aab812f5-f629-4ca6-9abb-b0c6414430f2 {"id":"aab812f5-f629-4ca6-9abb-b0c6414430f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752657} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14607a1 aab812f5-f629-4ca6-9abb-b0c6414430f2 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114296146984967,"min-table-barrier-ts":457114297195560966,"admin-job-type":0} /tidb/cdc/default/default/task/position/aab812f5-f629-4ca6-9abb-b0c6414430f2/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c6b8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114296146984967 MinTableBarrierTs:457114297195560966 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aab812f5-f629-4ca6-9abb-b0c6414430f2 {"id":"aab812f5-f629-4ca6-9abb-b0c6414430f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752657} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14607a1 aab812f5-f629-4ca6-9abb-b0c6414430f2 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114296146984967,"min-table-barrier-ts":457114297195560966,"admin-job-type":0} /tidb/cdc/default/default/task/position/aab812f5-f629-4ca6-9abb-b0c6414430f2/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c6b8c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114296146984967 MinTableBarrierTs:457114297195560966 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114296146984967, checkpointTs: 457114296146984967, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aab812f5-f629-4ca6-9abb-b0c6414430f2 {"id":"aab812f5-f629-4ca6-9abb-b0c6414430f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752657} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14607a1 aab812f5-f629-4ca6-9abb-b0c6414430f2 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114296146984967,"min-table-barrier-ts":457114297195560966,"admin-job-type":0} /tidb/cdc/default/default/task/position/aab812f5-f629-4ca6-9abb-b0c6414430f2/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' + set +x [2025/04/04 15:44:21.564 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5301.out cli changefeed pause -c test Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 15-th time, retry later + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6456.out cli changefeed resume -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 39-th check, retry later "is-owner": true, run task successfully PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 5-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:23 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e76486c1-8609-43c8-ad0c-b4f6f2738a60 {"id":"e76486c1-8609-43c8-ad0c-b4f6f2738a60","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752660} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d390e2 e76486c1-8609-43c8-ad0c-b4f6f2738a60 /tidb/cdc/default/default/upstream/7489360596269419294 {"id":7489360596269419294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e76486c1-8609-43c8-ad0c-b4f6f2738a60 {"id":"e76486c1-8609-43c8-ad0c-b4f6f2738a60","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752660} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d390e2 e76486c1-8609-43c8-ad0c-b4f6f2738a60 /tidb/cdc/default/default/upstream/7489360596269419294 {"id":7489360596269419294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e76486c1-8609-43c8-ad0c-b4f6f2738a60 {"id":"e76486c1-8609-43c8-ad0c-b4f6f2738a60","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752660} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d390e2 e76486c1-8609-43c8-ad0c-b4f6f2738a60 /tidb/cdc/default/default/upstream/7489360596269419294 {"id":7489360596269419294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x Create changefeed successfully! ID: 778e8616-b91c-425b-b5ac-22c47be29d19 Info: {"upstream_id":7489360596269419294,"namespace":"default","id":"778e8616-b91c-425b-b5ac-22c47be29d19","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:23.363188237+08:00","start_ts":457114298162348035,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114298162348035,"checkpoint_ts":457114298162348035,"checkpoint_time":"2025-04-04 15:44:23.278"} owner pid: 3318 owner id aab812f5-f629-4ca6-9abb-b0c6414430f2 process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v 'aab812f5-f629-4ca6-9abb-b0c6414430f2' /tidb/cdc/default/__cdc_meta__/owner/223195ffc14607a1 run task successfully process status: S cdc.test cli capture list 2>&1 | grep '"is-owner": true' + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5345.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=false' --no-confirm ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 16-th time, retry later split_and_random_merge scale: 20 *************************** 1. row *************************** count(distinct region_id): 1 table ddl_manager.finish_mark not exists for 40-th check, retry later Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=true To:mysql://root@127.0.0.1:3306/?batch-dml-enable=false} [2025/04/04 15:44:23.571 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Update changefeed config successfully! ID: test Info: {"upstream_id":7489360574444162717,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=false","create_time":"2025-04-04T15:44:19.816146175+08:00","start_ts":457114297222561796,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114297628885017,"checkpoint_time":"2025-04-04 15:44:21.243"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' + echo '{ "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null }' { "upstream_id": 7489360347588497912, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114296990302212, "checkpoint_time": "2025-04-04 15:44:18.807", "error": null } ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360347588497912, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114296990302212, '"checkpoint_time":' '"2025-04-04' '15:44:18.807",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6526.out cli changefeed remove -c changefeed-initialize-error Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 457114298301022215 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff successfully + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5377.out cli changefeed resume -c test "is-owner": true, run task successfully test_expire_owner pass VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360639630816058 TiDB Cluster ID. tikv_gc_leader_uuid 657ff084b48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:9001, start at 2025-04-04 15:44:23.78426212 +0800 CST m=+2.439780038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.825 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.812 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.812 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 41-th check, retry later wait process cdc.test exit for 1-th time... Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:44:25.578 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff failed 17-th time, retry later wait process cdc.test exit for 2-th time... + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_cleanup_stale_tasks [Fri Apr 4 15:44:26 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.34963498.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... table ddl_manager.finish_mark not exists for 42-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5415.out cli changefeed pause -c test wait process cdc.test exit for 2-th time... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:44:27.583 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_nfs Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 18-th time, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:27 CST 2025] <<<<<< run test case changefeed_error success! >>>>>> + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10296.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 43-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5447.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' --no-confirm + set +x + tso='457114299417231361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114299417231361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114299417231361 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:44:29 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1033310335.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:29.589 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] check diff failed 19-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:29 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043a6b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114298506543111 MinTableBarrierTs:457114299554856966 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114298506543111,"min-table-barrier-ts":457114299817000965,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043a6b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114298506543111 MinTableBarrierTs:457114299554856966 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114298506543111,"min-table-barrier-ts":457114299817000965,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043a6b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114298506543111 MinTableBarrierTs:457114299554856966 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114298506543111, checkpointTs: 457114298506543111, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114298506543111,"min-table-barrier-ts":457114299817000965,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=false To:mysql://root@127.0.0.1:3306/?batch-dml-enable=true} Update changefeed config successfully! ID: test Info: {"upstream_id":7489360574444162717,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2025-04-04T15:44:19.816146175+08:00","start_ts":457114297222561796,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114297628885017,"checkpoint_time":"2025-04-04 15:44:21.243"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... *************************** 1. row *************************** count(distinct region_id): 20 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 2-th time, retry later "is-owner": true, run task successfully + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5495.out cli changefeed resume -c test [2025/04/04 15:44:31.596 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2025/04/04 15:44:31.602 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:31.603 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 44-th check, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 20-th time, retry later owner pid: 3501 owner id 71fd3898-5141-478c-b5a9-38d460275631 [Fri Apr 4 15:44:32 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.36143616.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:32 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114299417231361 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10391.out cli changefeed create --start-ts=457114299417231361 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360642715955812,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:33.050645127+08:00","start_ts":457114299417231361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114299417231361,"checkpoint_ts":457114299417231361,"checkpoint_time":"2025-04-04 15:44:28.065"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 3-th time, retry later + set +x table batch_update_to_no_batch.v1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360670764786281 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08bd240018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:4009, start at 2025-04-04 15:44:31.08469034 +0800 CST m=+2.054730201 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:31.132 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:31.128 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:31.128 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360670764786281 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08bd240018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:4009, start at 2025-04-04 15:44:31.08469034 +0800 CST m=+2.054730201 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:31.132 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:31.128 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:31.128 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360666789446843 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08bff0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:4073, start at 2025-04-04 15:44:31.244304591 +0800 CST m=+2.153266298 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:31.252 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:31.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:31.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 45-th check, retry later [2025/04/04 15:44:33.610 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2025/04/04 15:44:33.616 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:33.618 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114300172206087 : [2025/04/04 15:44:31.707 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=134] [jobID=135] [query="CREATE TABLE IF NOT EXISTS `finishmark` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114300172206087] [2025/04/04 15:44:33.625 +08:00] [INFO] [case.go:503] ["all tables synced"] [endTs=457114300172206087] check diff failed 21-th time, retry later + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 + '[' mysql == mysql ']' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360682078246556 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08e6680031 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:4118, start at 2025-04-04 15:44:33.701739762 +0800 CST m=+2.534489300 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:33.708 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:33.757 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:33.757 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360682078246556 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08e6680031 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:4118, start at 2025-04-04 15:44:33.701739762 +0800 CST m=+2.534489300 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:33.708 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:33.757 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:33.757 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360684756051129 TiDB Cluster ID. tikv_gc_leader_uuid 657ff08e668000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:4182, start at 2025-04-04 15:44:33.716155921 +0800 CST m=+2.454417422 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:33.767 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:33.740 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:33.740 +0800 All versions after safe point can be accessed. (DO NOT EDIT) split_and_random_merge scale: 40 *************************** 1. row *************************** count(distinct region_id): 1 [Fri Apr 4 15:44:35 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.54455447.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 46-th check, retry later table batch_update_to_no_batch.v1 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:35 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016df3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301127720972 MinTableBarrierTs:457114301127720972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301127720972,"min-table-barrier-ts":457114301127720972,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016df3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301127720972 MinTableBarrierTs:457114301127720972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301127720972,"min-table-barrier-ts":457114301127720972,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016df3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301127720972 MinTableBarrierTs:457114301127720972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301127720972,"min-table-barrier-ts":457114301127720972,"admin-job-type":0} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "71fd3898-5141-478c-b5a9-38d460275631" | grep id check diff failed 22-th time, retry later table test.t1 exists + sleep 10 "cluster-id": "default" "id": "fbe75249-5952-48a0-a9b8-9896dcbb039e", "cluster-id": "default" run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360682578228093 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0901880014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:7115, start at 2025-04-04 15:44:35.462980238 +0800 CST m=+2.653658663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:35.517 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:35.526 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:35.526 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360682578228093 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0901880014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:7115, start at 2025-04-04 15:44:35.462980238 +0800 CST m=+2.653658663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:35.517 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:35.526 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:35.526 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360686458852060 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0903d80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:7182, start at 2025-04-04 15:44:35.619364064 +0800 CST m=+2.698971744 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:35.627 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:35.624 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:35.624 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } build mysql_test successfully run all mysql test cases time="2025-04-04T15:44:34+08:00" level=info msg="running tests: [alter_table bigint composite_index concurrent_ddl date_formats drop mysql_replace]" time="2025-04-04T15:44:34+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0001ac000 {{} {} 0} {0 0} [0xc0001be080] {[]} 1 0xc0001b0000 false map[0xc0001be080:map[0xc0001be080:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" check diff successfully table ddl_manager.finish_mark not exists for 47-th check, retry later table batch_update_to_no_batch.v1 not exists for 3-th check, retry later [Fri Apr 4 15:44:37 CST 2025] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.54565458.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:38.232 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark1000] [2025/04/04 15:44:38.232 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark1000] check diff failed 23-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:38 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1792957-7f5d-4cfa-aed3-ee36538eda26 {"id":"f1792957-7f5d-4cfa-aed3-ee36538eda26","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752675} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc21161d8 f1792957-7f5d-4cfa-aed3-ee36538eda26 /tidb/cdc/default/default/upstream/7489360670764786281 {"id":7489360670764786281,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1792957-7f5d-4cfa-aed3-ee36538eda26 {"id":"f1792957-7f5d-4cfa-aed3-ee36538eda26","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752675} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc21161d8 f1792957-7f5d-4cfa-aed3-ee36538eda26 /tidb/cdc/default/default/upstream/7489360670764786281 {"id":7489360670764786281,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1792957-7f5d-4cfa-aed3-ee36538eda26 {"id":"f1792957-7f5d-4cfa-aed3-ee36538eda26","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752675} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc21161d8 f1792957-7f5d-4cfa-aed3-ee36538eda26 /tidb/cdc/default/default/upstream/7489360670764786281 {"id":7489360670764786281,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.5495.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ capture_id: fbe75249-5952-48a0-a9b8-9896dcbb039e Create changefeed successfully! ID: ff24c57f-e5d1-4605-9877-ae21e2f28b92 Info: {"upstream_id":7489360670764786281,"namespace":"default","id":"ff24c57f-e5d1-4605-9877-ae21e2f28b92","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:38.939610892+08:00","start_ts":457114302244454404,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114302244454404,"checkpoint_ts":457114302244454404,"checkpoint_time":"2025-04-04 15:44:38.850"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:44:39 CST 2025] <<<<<< START cdc server in consistent_replicate_nfs case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_nfs.84238425.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 48-th check, retry later table batch_update_to_no_batch.v1 exists table batch_update_to_no_batch.recover_and_insert not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:40 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d158c25-0dd7-441a-a04e-46d2e929760b {"id":"9d158c25-0dd7-441a-a04e-46d2e929760b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2218ed8 9d158c25-0dd7-441a-a04e-46d2e929760b /tidb/cdc/default/default/upstream/7489360682078246556 {"id":7489360682078246556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d158c25-0dd7-441a-a04e-46d2e929760b {"id":"9d158c25-0dd7-441a-a04e-46d2e929760b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2218ed8 9d158c25-0dd7-441a-a04e-46d2e929760b /tidb/cdc/default/default/upstream/7489360682078246556 {"id":7489360682078246556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' [2025/04/04 15:44:40.232 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2025/04/04 15:44:40.239 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark1000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:40.240 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d158c25-0dd7-441a-a04e-46d2e929760b {"id":"9d158c25-0dd7-441a-a04e-46d2e929760b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2218ed8 9d158c25-0dd7-441a-a04e-46d2e929760b /tidb/cdc/default/default/upstream/7489360682078246556 {"id":7489360682078246556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 49-th check, retry later table batch_update_to_no_batch.recover_and_insert not exists for 2-th check, retry later check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... *************************** 1. row *************************** count(distinct region_id): 40 0 [Fri Apr 4 15:44:41 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.37613763.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server3.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server3 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:44:41.840 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark2000] [2025/04/04 15:44:41.840 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark2000] start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:44:42.329 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2025/04/04 15:44:42.340 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark1000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:42.342 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:42 CST 2025] <<<<<< run test case clustered_index success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:42 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8c6200e-37c6-4116-832f-8c9cb4df2320 {"id":"c8c6200e-37c6-4116-832f-8c9cb4df2320","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752679} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc221e2d9 c8c6200e-37c6-4116-832f-8c9cb4df2320 /tidb/cdc/default/default/upstream/7489360682578228093 {"id":7489360682578228093,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8c6200e-37c6-4116-832f-8c9cb4df2320 {"id":"c8c6200e-37c6-4116-832f-8c9cb4df2320","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752679} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc221e2d9 c8c6200e-37c6-4116-832f-8c9cb4df2320 /tidb/cdc/default/default/upstream/7489360682578228093 {"id":7489360682578228093,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8c6200e-37c6-4116-832f-8c9cb4df2320 {"id":"c8c6200e-37c6-4116-832f-8c9cb4df2320","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752679} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc221e2d9 c8c6200e-37c6-4116-832f-8c9cb4df2320 /tidb/cdc/default/default/upstream/7489360682578228093 {"id":7489360682578228093,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x found first match line: 457114302531502083 : [2025/04/04 15:44:40.830 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=136] [jobID=137] [query="CREATE TABLE IF NOT EXISTS `finishmark1000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114302531502083] [2025/04/04 15:44:42.532 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark1000] [2025/04/04 15:44:42.532 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114302531502083] [2025/04/04 15:44:42.537 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:42.541 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:42.546 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:42.634 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] check diff failed 2-th time, retry later ***************** properties ***************** "fieldcount"="3" "requestdistribution"="uniform" "mysql.user"="root" "dotransactions"="false" "threadcount"="10" "operationcount"="0" "mysql.port"="4000" "insertproportion"="0" "updateproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "recordcount"="50000" "scanproportion"="0" "readallfields"="true" "mysql.db"="consistent_replicate_nfs" "readproportion"="0" ********************************************** table ddl_manager.finish_mark not exists for 50-th check, retry later table batch_update_to_no_batch.recover_and_insert not exists for 3-th check, retry later Verifying downstream PD is started... [2025/04/04 15:44:43.840 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] [2025/04/04 15:44:43.936 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark2000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:43.938 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114303444811780 : [2025/04/04 15:44:43.757 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=138] [jobID=139] [query="CREATE TABLE IF NOT EXISTS `finishmark2000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114303444811780] [2025/04/04 15:44:44.133 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark2000] [2025/04/04 15:44:44.133 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114303444811780] [2025/04/04 15:44:44.136 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:44.139 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:44.234 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:44.333 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_delay.t1 exists table processor_delay.t2 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:44 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ac990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301914152972 MinTableBarrierTs:457114301914152972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a08c072-3d46-4ed0-8c8c-90b26d3da806 {"id":"2a08c072-3d46-4ed0-8c8c-90b26d3da806","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752681} /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/__cdc_meta__/owner/223195ffc14608db 2a08c072-3d46-4ed0-8c8c-90b26d3da806 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301914152972,"min-table-barrier-ts":457114301914152972,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a08c072-3d46-4ed0-8c8c-90b26d3da806/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ac990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301914152972 MinTableBarrierTs:457114301914152972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a08c072-3d46-4ed0-8c8c-90b26d3da806 {"id":"2a08c072-3d46-4ed0-8c8c-90b26d3da806","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752681} /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/__cdc_meta__/owner/223195ffc14608db 2a08c072-3d46-4ed0-8c8c-90b26d3da806 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301914152972,"min-table-barrier-ts":457114301914152972,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a08c072-3d46-4ed0-8c8c-90b26d3da806/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ac990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114301914152972 MinTableBarrierTs:457114301914152972 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a08c072-3d46-4ed0-8c8c-90b26d3da806 {"id":"2a08c072-3d46-4ed0-8c8c-90b26d3da806","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752681} /tidb/cdc/default/__cdc_meta__/capture/71fd3898-5141-478c-b5a9-38d460275631 {"id":"71fd3898-5141-478c-b5a9-38d460275631","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752666} /tidb/cdc/default/__cdc_meta__/capture/fbe75249-5952-48a0-a9b8-9896dcbb039e {"id":"fbe75249-5952-48a0-a9b8-9896dcbb039e","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752672} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460811 71fd3898-5141-478c-b5a9-38d460275631 /tidb/cdc/default/__cdc_meta__/owner/223195ffc146085a fbe75249-5952-48a0-a9b8-9896dcbb039e /tidb/cdc/default/__cdc_meta__/owner/223195ffc14608db 2a08c072-3d46-4ed0-8c8c-90b26d3da806 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114301914152972,"min-table-barrier-ts":457114301914152972,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a08c072-3d46-4ed0-8c8c-90b26d3da806/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/71fd3898-5141-478c-b5a9-38d460275631/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/fbe75249-5952-48a0-a9b8-9896dcbb039e/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' run task failed 1-th time, retry later check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 51-th check, retry later table batch_update_to_no_batch.recover_and_insert exists table batch_update_to_no_batch.finish_mark not exists for 1-th check, retry later split_and_random_merge scale: 80 *************************** 1. row *************************** count(distinct region_id): 20 + run_sql 'SELECT * from test.t1' 127.0.0.1 3306 + check_contains 'b: 123456' + check_contains 'b: aaaaa' + check_contains 'c: ab\\\\c' + check_contains 'c: ab\\c' + stop_tidb_cluster Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t2 not exists for 2-th check, retry later cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' run task failed 2-th time, retry later [2025/04/04 15:44:47.541 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark3000] [2025/04/04 15:44:47.541 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark3000] table ddl_manager.finish_mark not exists for 52-th check, retry later table batch_update_to_no_batch.finish_mark not exists for 2-th check, retry later check diff failed 4-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t2 exists table processor_delay.t3 not exists for 1-th check, retry later [2025/04/04 15:44:49.541 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] table ddl_manager.finish_mark not exists for 53-th check, retry later table batch_update_to_no_batch.finish_mark not exists for 3-th check, retry later check diff successfully [2025/04/04 15:44:49.640 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark3000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:49.642 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table batch_add_table.finish_mark not exists for 1-th check, retry later table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 54-th check, retry later table batch_update_to_no_batch.finish_mark not exists for 4-th check, retry later cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' table batch_add_table.finish_mark not exists for 2-th check, retry later [2025/04/04 15:44:51.930 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2025/04/04 15:44:51.939 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark3000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:51.940 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] [2025/04/04 15:44:52.044 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark4000] [2025/04/04 15:44:52.044 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark4000] found first match line: 457114304864583688 : [2025/04/04 15:44:49.930 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=140] [jobID=141] [query="CREATE TABLE IF NOT EXISTS `finishmark3000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114304864583688] [2025/04/04 15:44:52.142 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark3000] [2025/04/04 15:44:52.142 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114304864583688] [2025/04/04 15:44:52.144 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:52.233 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:52.238 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:52.246 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later *************************** 1. row *************************** count(distinct region_id): 80 "is-owner": true, run task successfully nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360749637881821 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a038c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:7504, start at 2025-04-04 15:44:51.940895352 +0800 CST m=+3.616317336 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:51.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:51.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:51.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360749637881821 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a038c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:7504, start at 2025-04-04 15:44:51.940895352 +0800 CST m=+3.616317336 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:51.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:51.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:51.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360756970976519 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a03800005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:7572, start at 2025-04-04 15:44:51.942193168 +0800 CST m=+3.510557153 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:51.956 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:51.936 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:51.936 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } INSERT - Takes(s): 10.0, Count: 15537, OPS: 1555.2, Avg(us): 6410, Min(us): 862, Max(us): 100135, 95th(us): 54000, 99th(us): 83000 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 55-th check, retry later table batch_update_to_no_batch.finish_mark exists check diff successfully table batch_add_table.finish_mark not exists for 3-th check, retry later [2025/04/04 15:44:54.044 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [2025/04/04 15:44:54.136 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark4000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:54.138 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 1-th time... found first match line: 457114305886945286 : [2025/04/04 15:44:52.930 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=142] [jobID=143] [query="CREATE TABLE IF NOT EXISTS `finishmark4000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114305886945286] [2025/04/04 15:44:54.352 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark4000] [2025/04/04 15:44:54.352 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114305886945286] [2025/04/04 15:44:54.354 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:54.434 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:54.442 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:44:54.532 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8870.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:52 CST 2025] <<<<<< run test case ddl_puller_lag success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:55 CST 2025] <<<<<< run test case batch_update_to_no_batch success! >>>>>> nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ./t/alter_table.test: ok! 64 test cases passed, take time 17.8507992 s time="2025-04-04T15:44:53+08:00" level=info msg="run test [alter_table] ok" time="2025-04-04T15:44:53+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0003122d0 {{} {} 0} {0 0} [0xc000344b80] {[]} 1 0xc000300930 false map[0xc000344b80:map[0xc000344b80:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 56-th check, retry later table batch_add_table.finish_mark not exists for 4-th check, retry later + set +x + tso='457114306514518017 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114306514518017 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:56 CST 2025] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.89048906.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t6 exists table processor_delay.t7 not exists for 1-th check, retry later [2025/04/04 15:44:56.947 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark5000] [2025/04/04 15:44:56.948 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark5000] nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 57-th check, retry later table region_merge.t1 exists check diff failed 1-th time, retry later table batch_add_table.finish_mark not exists for 5-th check, retry later ***************** properties ***************** "operationcount"="0" "mysql.port"="4000" "readproportion"="0" "threadcount"="2" "insertproportion"="0" "updateproportion"="0" "recordcount"="10" "mysql.user"="root" "dotransactions"="false" "workload"="core" "mysql.host"="127.0.0.1" "scanproportion"="0" "requestdistribution"="uniform" "mysql.db"="testSync" "readallfields"="true" ********************************************** Run finished, takes 5.628451ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2770.0, Avg(us): 1034, Min(us): 569, Max(us): 2068, 95th(us): 3000, 99th(us): 3000 + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later [2025/04/04 15:44:58.948 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2025/04/04 15:44:58.955 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark5000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:44:58.957 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] Verifying downstream PD is started... empty select id, val from test.availability1 where id=1 run task successfully test_owner_cleanup_stale_tasks pass found first match line: 457114307224141829 : [2025/04/04 15:44:58.857 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=144] [jobID=145] [query="CREATE TABLE IF NOT EXISTS `finishmark5000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114307224141829] [2025/04/04 15:44:59.344 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark5000] [2025/04/04 15:44:59.344 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114307224141829] [2025/04/04 15:44:59.347 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:59.431 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:44:59.438 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:44:59.443 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_manager.finish_mark not exists for 58-th check, retry later wait process cdc.test exit for 1-th time... check diff successfully table batch_add_table.finish_mark not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:59 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b219bee0-7ecd-4ba2-a561-76fc0fbf506d {"id":"b219bee0-7ecd-4ba2-a561-76fc0fbf506d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752697} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25a93ea b219bee0-7ecd-4ba2-a561-76fc0fbf506d /tidb/cdc/default/default/upstream/7489360749637881821 {"id":7489360749637881821,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b219bee0-7ecd-4ba2-a561-76fc0fbf506d {"id":"b219bee0-7ecd-4ba2-a561-76fc0fbf506d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752697} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25a93ea b219bee0-7ecd-4ba2-a561-76fc0fbf506d /tidb/cdc/default/default/upstream/7489360749637881821 {"id":7489360749637881821,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b219bee0-7ecd-4ba2-a561-76fc0fbf506d {"id":"b219bee0-7ecd-4ba2-a561-76fc0fbf506d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752697} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25a93ea b219bee0-7ecd-4ba2-a561-76fc0fbf506d /tidb/cdc/default/default/upstream/7489360749637881821 {"id":7489360749637881821,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8960.out cli changefeed create --start-ts=457114306514518017 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: 72ec5cf1-2f03-47e3-ab76-f9fa327e88ac Info: {"upstream_id":7489360749637881821,"namespace":"default","id":"72ec5cf1-2f03-47e3-ab76-f9fa327e88ac","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:00.217108983+08:00","start_ts":457114306514518017,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114306514518017,"checkpoint_ts":457114306514518017,"checkpoint_time":"2025-04-04 15:44:55.139"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... table processor_delay.t8 exists table processor_delay.t9 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_retryable_error [Fri Apr 4 15:45:00 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/capture/capture-campaign-compacted-error=1*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.39994001.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:01 CST 2025] <<<<<< run test case region_merge success! >>>>>> TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:45:01.433 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark6000] [2025/04/04 15:45:01.433 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark6000] table ddl_manager.finish_mark not exists for 59-th check, retry later table batch_add_table.finish_mark not exists for 7-th check, retry later + set +x start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... check diff failed 5-th time, retry later Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... INSERT - Takes(s): 20.0, Count: 25889, OPS: 1295.1, Avg(us): 7696, Min(us): 862, Max(us): 100135, 95th(us): 75000, 99th(us): 84000 check diff successfully [2025/04/04 15:45:03.433 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2025/04/04 15:45:03.441 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark6000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:03.443 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040aef30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114307432284166 MinTableBarrierTs:457114308743266311 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114307432284166,"min-table-barrier-ts":457114308743266311,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040aef30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114307432284166 MinTableBarrierTs:457114308743266311 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114307432284166,"min-ta+ grep -q 'failed to get info:' ble-barrier-ts":457114308743266311,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040aef30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114307432284166 MinTableBarrierTs:457114308743266311 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114307432284166, checkpointTs: 457114307432284166, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114307432284166,"min-table-barrier-ts":457114308743266311,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table ddl_manager.finish_mark not exists for 60-th check, retry later table batch_add_table.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table processor_delay.t10 exists table processor_delay.t11 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_resume_with_checkpoint_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... "is-owner": true, run task successfully wait process cdc.test exit for 3-th time... [2025/04/04 15:45:05.738 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark7000] [2025/04/04 15:45:05.738 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark7000] start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 61-th check, retry later table batch_add_table.finish_mark not exists for 9-th check, retry later [2025/04/04 15:45:05.943 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2025/04/04 15:45:05.950 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark6000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:05.951 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:45:05 CST 2025] <<<<<< run test case changefeed_pause_resume success! >>>>>> found first match line: 457114308587028487 : [2025/04/04 15:45:03.957 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=146] [jobID=147] [query="CREATE TABLE IF NOT EXISTS `finishmark6000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114308587028487] [2025/04/04 15:45:06.353 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark6000] [2025/04/04 15:45:06.353 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114308587028487] [2025/04/04 15:45:06.357 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:06.360 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:06.438 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:06.443 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] owner pid: 4004 owner id 194727e7-2384-4dc6-a4de-8db1b639994b [Fri Apr 4 15:45:06 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/owner-run-with-error=1*return(true);github.com/pingcap/tiflow/cdc/capture/capture-resign-failed=1*return(true)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.41084110.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server2 --cluster-id default --addr 127.0.0.1:8301 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ./t/bigint.test: ok! 138 test cases passed, take time 12.843351899 s time="2025-04-04T15:45:06+08:00" level=info msg="run test [bigint] ok" time="2025-04-04T15:45:06+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000312278 {{} {} 0} {0 0} [0xc0002dec00] {[]} 1 0xc0003008c0 false map[0xc0002dec00:map[0xc0002dec00:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360810212239345 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ad4840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:4712, start at 2025-04-04 15:45:05.316043595 +0800 CST m=+2.208421036 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:05.324 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:05.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:05.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360810212239345 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ad4840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:4712, start at 2025-04-04 15:45:05.316043595 +0800 CST m=+2.208421036 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:05.324 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:05.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:05.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360818559768394 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ad614000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-htn1z-434rm-lwx34, pid:4779, start at 2025-04-04 15:45:05.431521351 +0800 CST m=+2.248145057 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:05.439 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:05.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:05.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 62-th check, retry later table batch_add_table.finish_mark not exists for 10-th check, retry later [2025/04/04 15:45:07.738 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2025/04/04 15:45:07.747 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark7000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:07.748 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114309661818881 : [2025/04/04 15:45:07.957 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=148] [jobID=149] [query="CREATE TABLE IF NOT EXISTS `finishmark7000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114309661818881] [2025/04/04 15:45:08.147 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark7000] [2025/04/04 15:45:08.147 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114309661818881] [2025/04/04 15:45:08.233 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:08.239 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_sequence.finish_mark not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:45:08.329 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:08.337 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] start tidb cluster in /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ./t/composite_index.test: ok! 12 test cases passed, take time 1.540386949 s time="2025-04-04T15:45:08+08:00" level=info msg="run test [composite_index] ok" time="2025-04-04T15:45:08+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000fc3b8 {{} {} 0} {0 0} [0xc00027e780] {[]} 1 0xc00003f1f0 false map[0xc00027e780:map[0xc00027e780:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later [Fri Apr 4 15:45:09 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.60206022.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 63-th check, retry later table batch_add_table.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:09 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003801e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310053724171 MinTableBarrierTs:457114310316130315 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/capture/8fe815be-edea-461e-b714-45ff1f402d57 {"id":"8fe815be-edea-461e-b714-45ff1f402d57","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752707} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310053724171,"min-table-barrier-ts":457114310316130315,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8fe815be-edea-461e-b714-45ff1f402d57/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003801e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310053724171 MinTableBarrierTs:457114310316130315 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/capture/8fe815be-edea-461e-b714-45ff1f402d57 {"id":"8fe815be-edea-461e-b714-45ff1f402d57","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752707} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310053724171,"min-table-barrier-ts":457114310316130315,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8fe815be-edea-461e-b714-45ff1f402d57/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003801e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310053724171 MinTableBarrierTs:457114310316130315 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/194727e7-2384-4dc6-a4de-8db1b639994b {"id":"194727e7-2384-4dc6-a4de-8db1b639994b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752701} /tidb/cdc/default/__cdc_meta__/capture/8fe815be-edea-461e-b714-45ff1f402d57 {"id":"8fe815be-edea-461e-b714-45ff1f402d57","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752707} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460998 194727e7-2384-4dc6-a4de-8db1b639994b /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310053724171,"min-table-barrier-ts":457114310316130315,"admin-job-type":0} /tidb/cdc/default/default/task/position/194727e7-2384-4dc6-a4de-8db1b639994b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8fe815be-edea-461e-b714-45ff1f402d57/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "194727e7-2384-4dc6-a4de-8db1b639994b" | grep id table ddl_sequence.finish_mark not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ./t/concurrent_ddl.test: ok! 8 test cases passed, take time 1.020888517 s time="2025-04-04T15:45:10+08:00" level=info msg="run test [concurrent_ddl] ok" time="2025-04-04T15:45:10+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0005a2020 {{} {} 0} {0 0} [0xc000534200] {[]} 1 0xc0005960e0 false map[0xc000534200:map[0xc000534200:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360639630816058 TiDB Cluster ID. tikv_gc_leader_uuid 657ff084b48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:9001, start at 2025-04-04 15:44:23.78426212 +0800 CST m=+2.439780038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.825 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.812 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.812 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... "cluster-id": "default" "id": "8fe815be-edea-461e-b714-45ff1f402d57", "cluster-id": "default" run task successfully table ddl_manager.finish_mark not exists for 64-th check, retry later [2025/04/04 15:45:11.430 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark8000] [2025/04/04 15:45:11.430 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark8000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_sequence.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 12-th check, retry later capture_id: 8fe815be-edea-461e-b714-45ff1f402d57 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 35 100 35 0 0 35000 0 --:--:-- --:--:-- --:--:-- 35000 { "status": true, "message": "" }ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '194727e7-2384-4dc6-a4de-8db1b639994b' 194727e7-2384-4dc6-a4de-8db1b639994b run task successfully ps -C cdc.test -o pid= | awk '{print $1}' | wc -l | grep 2 2 run task successfully test_owner_retryable_error pass table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:12 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.6087.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: 23e5ff92-a083-48d4-9543-69233f834101 Info: {"upstream_id":7489360810212239345,"namespace":"default","id":"23e5ff92-a083-48d4-9543-69233f834101","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:13.072227942+08:00","start_ts":457114311182254084,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114311182254084,"checkpoint_ts":457114311182254084,"checkpoint_time":"2025-04-04 15:45:12.945"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... [2025/04/04 15:45:13.431 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2025/04/04 15:45:13.439 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark8000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:13.440 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) INSERT - Takes(s): 30.0, Count: 38812, OPS: 1294.1, Avg(us): 7698, Min(us): 843, Max(us): 100135, 95th(us): 75000, 99th(us): 85000 wait process cdc.test exit for 2-th time... found first match line: 457114311024967681 : [2025/04/04 15:45:12.958 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=150] [jobID=151] [query="CREATE TABLE IF NOT EXISTS `finishmark8000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114311024967681] [2025/04/04 15:45:13.844 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark8000] [2025/04/04 15:45:13.844 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114311024967681] [2025/04/04 15:45:13.847 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:13.850 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:13.854 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:13.859 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_sequence.finish_mark not exists for 4-th check, retry later table batch_add_table.finish_mark exists check diff successfully table ddl_manager.finish_mark not exists for 65-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/common_1 Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_gap_between_watch_capture [Fri Apr 4 15:45:14 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.42724274.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360848791361006 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b43f80035 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:6536, start at 2025-04-04 15:45:12.472647905 +0800 CST m=+2.506364170 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:12.530 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:12.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:12.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360848791361006 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b43f80035 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:6536, start at 2025-04-04 15:45:12.472647905 +0800 CST m=+2.506364170 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:12.530 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:12.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:12.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360848030455189 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b45340009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:6595, start at 2025-04-04 15:45:12.539647702 +0800 CST m=+2.493428767 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:12.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:12.542 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:12.542 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit table ddl_manager.finish_mark not exists for 66-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:45:16.036 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark9000] [2025/04/04 15:45:16.036 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark9000] table ddl_sequence.finish_mark not exists for 5-th check, retry later [Fri Apr 4 15:45:16 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.78247826.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ./t/date_formats.test: ok! 99 test cases passed, take time 5.267252796 s time="2025-04-04T15:45:15+08:00" level=info msg="run test [date_formats] ok" time="2025-04-04T15:45:15+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0005a22a8 {{} {} 0} {0 0} [0xc0002d5e80] {[]} 1 0xc000596ee0 false map[0xc0002d5e80:map[0xc0002d5e80:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 exists [Fri Apr 4 15:45:16 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.62106212.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4002 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004114990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310840156171 MinTableBarrierTs:457114311102300172 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310840156171,"min-table-barrier-ts":457114311102300172,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004114990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310840156171 MinTableBarrierTs:457114311102300172 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310840156171,"min-table-barrier-ts":457114311102300172,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004114990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114310840156171 MinTableBarrierTs:457114311102300172 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14609e9 8fe815be-edea-461e-b714-45ff1f402d57 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114310840156171,"min-table-barrier-ts":457114311102300172,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 67-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360863193495274 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b7950000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:6557, start at 2025-04-04 15:45:15.881421302 +0800 CST m=+2.342089354 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:15.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:15.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:15.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360863193495274 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b7950000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:6557, start at 2025-04-04 15:45:15.881421302 +0800 CST m=+2.342089354 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:15.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:15.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:15.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360854701292911 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b7a64000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xb2bm-8cz4s-8mtgw, pid:6620, start at 2025-04-04 15:45:15.975350787 +0800 CST m=+2.340423461 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:15.983 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:15.980 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:15.980 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:45:18.036 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2025/04/04 15:45:18.043 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark9000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:18.045 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table ddl_sequence.finish_mark exists check diff successfully found first match line: 457114312204615681 : [2025/04/04 15:45:16.958 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=152] [jobID=153] [query="CREATE TABLE IF NOT EXISTS `finishmark9000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114312204615681] [2025/04/04 15:45:18.556 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark9000] [2025/04/04 15:45:18.556 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114312204615681] [2025/04/04 15:45:18.630 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:18.635 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:18.641 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:18.730 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] wait process cdc.test exit for 1-th time... table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f64dd36a-1839-4c93-b2ba-8d2a42003ac2 {"id":"f64dd36a-1839-4c93-b2ba-8d2a42003ac2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752716} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b730d7 f64dd36a-1839-4c93-b2ba-8d2a42003ac2 /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f64dd36a-1839-4c93-b2ba-8d2a42003ac2 {"id":"f64dd36a-1839-4c93-b2ba-8d2a42003ac2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752716} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b730d7 f64dd36a-1839-4c93-b2ba-8d2a42003ac2 /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f64dd36a-1839-4c93-b2ba-8d2a42003ac2 {"id":"f64dd36a-1839-4c93-b2ba-8d2a42003ac2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752716} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b730d7 f64dd36a-1839-4c93-b2ba-8d2a42003ac2 /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.7886.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-only-block-related-table table ddl_manager.finish_mark not exists for 68-th check, retry later [Fri Apr 4 15:45:19 CST 2025] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.79207922.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7489360848791361006,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:45:19.922664266+08:00","start_ts":457114312991571970,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114312991571970,"checkpoint_ts":457114312991571970,"checkpoint_time":"2025-04-04 15:45:19.847"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test cli capture list 2>&1 | grep '"is-owner": true' run task failed 2-th time, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:45:20 CST 2025] <<<<<< run test case ddl_sequence success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/23e5ff92-a083-48d4-9543-69233f834101 {UpstreamID:7489360810212239345 Namespace:default ID:23e5ff92-a083-48d4-9543-69233f834101 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:13.072227942 +0800 CST StartTs:457114311182254084 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cff3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114311208730630} {CheckpointTs:457114312807546888 MinTableBarrierTs:457114312807546888 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/127e2466-607f-4148-8500-d4bb031737de {"id":"127e2466-607f-4148-8500-d4bb031737de","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965f55 127e2466-607f-4148-8500-d4bb031737de /tidb/cdc/default/default/changefeed/info/23e5ff92-a083-48d4-9543-69233f834101 {"upstream-id":7489360810212239345,"namespace":"default","changefeed-id":"23e5ff92-a083-48d4-9543-69233f834101","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:13.072227942+08:00","start-ts":457114311182254084,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114311208730630} /tidb/cdc/default/default/changefeed/status/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":457114312807546888,"min-table-barrier-ts":457114312807546888,"admin-job-type":0} /tidb/cdc/default/default/task/position/127e2466-607f-4148-8500-d4bb031737de/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e3347097-3b83-430f-8551-de9b4d8ba5d0/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/23e5ff92-a083-48d4-9543-69233f834101 {UpstreamID:7489360810212239345 Namespace:default ID:23e5ff92-a083-48d4-9543-69233f834101 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:13.072227942 +0800 CST StartTs:457114311182254084 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cff3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114311208730630} {CheckpointTs:457114312807546888 MinTableBarrierTs:457114312807546888 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/127e2466-607f-4148-8500-d4bb031737de {"id":"127e2466-607f-4148-8500-d4bb031737de","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965f55 127e2466-607f-4148-8500-d4bb031737de /tidb/cdc/default/default/changefeed/info/23e5ff92-a083-48d4-9543-69233f834101 {"upstream-id":7489360810212239345,"namespace":"default","changefeed-id":"23e5ff92-a083-48d4-9543-69233f834101","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:13.072227942+08:00","start-ts":457114311182254084,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114311208730630} /tidb/cdc/default/default/changefeed/status/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":457114312807546888,"min-table-barrier-ts":457114312807546888,"admin-job-type":0} /tidb/cdc/default/default/task/position/127e2466-607f-4148-8500-d4bb031737de/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e3347097-3b83-430f-8551-de9b4d8ba5d0/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/23e5ff92-a083-48d4-9543-69233f834101 {UpstreamID:7489360810212239345 Namespace:default ID:23e5ff92-a083-48d4-9543-69233f834101 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:13.072227942 +0800 CST StartTs:457114311182254084 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cff3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114311208730630} {CheckpointTs:457114312807546888 MinTableBarrierTs:457114312807546888 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/127e2466-607f-4148-8500-d4bb031737de {"id":"127e2466-607f-4148-8500-d4bb031737de","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/e3347097-3b83-430f-8551-de9b4d8ba5d0 {"id":"e3347097-3b83-430f-8551-de9b4d8ba5d0","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965ed8 e3347097-3b83-430f-8551-de9b4d8ba5d0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2965f55 127e2466-607f-4148-8500-d4bb031737de /tidb/cdc/default/default/changefeed/info/23e5ff92-a083-48d4-9543-69233f834101 {"upstream-id":7489360810212239345,"namespace":"default","changefeed-id":"23e5ff92-a083-48d4-9543-69233f834101","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:13.072227942+08:00","start-ts":457114311182254084,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114311208730630} /tidb/cdc/default/default/changefeed/status/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":457114312807546888,"min-table-barrier-ts":457114312807546888,"admin-job-type":0} /tidb/cdc/default/default/task/position/127e2466-607f-4148-8500-d4bb031737de/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e3347097-3b83-430f-8551-de9b4d8ba5d0/23e5ff92-a083-48d4-9543-69233f834101 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360810212239345 {"id":7489360810212239345,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:45:21.139 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark10000] [2025/04/04 15:45:21.139 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark10000] + set +x table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 69-th check, retry later Run finished, takes 38.02772281s INSERT - Takes(s): 38.0, Count: 49998, OPS: 1315.1, Avg(us): 7565, Min(us): 843, Max(us): 100135, 95th(us): 75000, 99th(us): 85000 table consistent_replicate_nfs.usertable exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... [2025/04/04 15:45:23.139 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:23 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f64355-af33-45e2-88e2-7fa377733a8d {"id":"47f64355-af33-45e2-88e2-7fa377733a8d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c134d7 47f64355-af33-45e2-88e2-7fa377733a8d /tidb/cdc/default/default/upstream/7489360863193495274 {"id":7489360863193495274,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f64355-af33-45e2-88e2-7fa377733a8d {"id":"47f64355-af33-45e2-88e2-7fa377733a8d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c134d7 47f64355-af33-45e2-88e2-7fa377733a8d /tidb/cdc/default/default/upstream/7489360863193495274 {"id":7489360863193495274,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f64355-af33-45e2-88e2-7fa377733a8d {"id":"47f64355-af33-45e2-88e2-7fa377733a8d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c134d7 47f64355-af33-45e2-88e2-7fa377733a8d /tidb/cdc/default/default/upstream/7489360863193495274 {"id":7489360863193495274,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:45:23.234 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark10000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:23.236 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 3-th time... table test.table1 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later found first match line: 457114313511141389 : [2025/04/04 15:45:22.030 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=154] [jobID=155] [query="CREATE TABLE IF NOT EXISTS `finishmark10000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114313511141389] [2025/04/04 15:45:23.847 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark10000] [2025/04/04 15:45:23.847 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114313511141389] [2025/04/04 15:45:23.849 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:23.930 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:23.935 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:23.940 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit table ddl_manager.finish_mark not exists for 70-th check, retry later table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully cdc.test cli capture list 2>&1 | grep '"is-owner": true' killing cdc1: 6025 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later run task failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360886006559165 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bdae00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:6669, start at 2025-04-04 15:45:22.113510161 +0800 CST m=+2.135861517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:22.120 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:22.104 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:22.104 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360886006559165 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bdae00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:6669, start at 2025-04-04 15:45:22.113510161 +0800 CST m=+2.135861517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:22.120 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:22.104 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:22.104 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360889100505789 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bdc6c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9jjm4-656tq-hjz7q, pid:6744, start at 2025-04-04 15:45:22.205530609 +0800 CST m=+2.153884075 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:22.213 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:22.203 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:22.203 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/common_1/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/common_1/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/common_1/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:45:25.531 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark11000] [2025/04/04 15:45:25.531 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark11000] table test.table1 exists table test.table2 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 71-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ./t/drop.test: ok! 48 test cases passed, take time 8.665583502 s time="2025-04-04T15:45:25+08:00" level=info msg="run test [drop] ok" time="2025-04-04T15:45:25+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000684190 {{} {} 0} {0 0} [0xc0001e7300] {[]} 1 0xc000385030 false map[0xc0001e7300:map[0xc0001e7300:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" ./t/mysql_replace.test: ok! 16 test cases passed, take time 1.044129287 s time="2025-04-04T15:45:26+08:00" level=info msg="run test [mysql_replace] ok" Great, All tests passed mysqltest end cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later table test.finish_mark not exists for 1-th check, retry later [2025/04/04 15:45:27.532 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2025/04/04 15:45:27.539 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark11000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:27.542 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.table2 exists check diff failed 1-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 72-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) found first match line: 457114314904698881 : [2025/04/04 15:45:28.130 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=156] [jobID=157] [query="CREATE TABLE IF NOT EXISTS `finishmark11000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114314904698881] [2025/04/04 15:45:28.444 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark11000] [2025/04/04 15:45:28.444 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114314904698881] [2025/04/04 15:45:28.532 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:28.536 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:28.544 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:28.632 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.8007.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check_changefeed_state http://127.0.0.1:2379 aa5a6788-a843-4215-9680-ccde3119ce27 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=aa5a6788-a843-4215-9680-ccde3119ce27 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c aa5a6788-a843-4215-9680-ccde3119ce27 -s + info='{ "upstream_id": 7489360450719664448, "namespace": "default", "id": "aa5a6788-a843-4215-9680-ccde3119ce27", "state": "finished", "checkpoint_tso": 457114312233713666, "checkpoint_time": "2025-04-04 15:45:16.956", "error": null }' + echo '{ "upstream_id": 7489360450719664448, "namespace": "default", "id": "aa5a6788-a843-4215-9680-ccde3119ce27", "state": "finished", "checkpoint_tso": 457114312233713666, "checkpoint_time": "2025-04-04 15:45:16.956", "error": null }' { "upstream_id": 7489360450719664448, "namespace": "default", "id": "aa5a6788-a843-4215-9680-ccde3119ce27", "state": "finished", "checkpoint_tso": 457114312233713666, "checkpoint_time": "2025-04-04 15:45:16.956", "error": null } ++ echo '{' '"upstream_id":' 7489360450719664448, '"namespace":' '"default",' '"id":' '"aa5a6788-a843-4215-9680-ccde3119ce27",' '"state":' '"finished",' '"checkpoint_tso":' 457114312233713666, '"checkpoint_time":' '"2025-04-04' '15:45:16.956",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7489360450719664448, '"namespace":' '"default",' '"id":' '"aa5a6788-a843-4215-9680-ccde3119ce27",' '"state":' '"finished",' '"checkpoint_tso":' 457114312233713666, '"checkpoint_time":' '"2025-04-04' '15:45:16.956",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:45:25 CST 2025] <<<<<< run test case changefeed_finish success! >>>>>> check diff successfully table ddl_only_block_related_table.finish_mark exists table ddl_manager.finish_mark not exists for 73-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360913722737017 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c3340000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:9826, start at 2025-04-04 15:45:27.790579938 +0800 CST m=+2.349277903 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:27.836 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:27.810 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:27.810 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360913722737017 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c3340000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:9826, start at 2025-04-04 15:45:27.790579938 +0800 CST m=+2.349277903 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:27.836 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:27.810 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:27.810 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360912385470048 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c34bc0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:9889, start at 2025-04-04 15:45:27.862495183 +0800 CST m=+2.329067487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:27.871 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:27.855 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:27.855 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:45:29.641 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark12000] [2025/04/04 15:45:29.641 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark12000] wait process 7829 exit for 1-th time... wait process 7829 exit for 2-th time... + set +x + tso='457114315326488577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114315326488577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:45:30 CST 2025] <<<<<< START cdc server in common_1 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.80488050.out server --log-file /tmp/tidb_cdc_test/common_1/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/common_1/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Resume changefeed with checkpointTs3 457114315721277445 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360639630816058 TiDB Cluster ID. tikv_gc_leader_uuid 657ff084b48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:9001, start at 2025-04-04 15:44:23.78426212 +0800 CST m=+2.439780038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.825 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.812 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.812 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... table test.table5 not exists for 1-th check, retry later table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later cdc.test cli capture list 2>&1 | grep '"is-owner": true' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7829) - No such process wait process 7829 exit for 3-th time... process 7829 already exit [Fri Apr 4 15:45:31 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.79727974.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later table test.finish_mark not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 74-th check, retry later [2025/04/04 15:45:31.641 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [2025/04/04 15:45:31.648 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark12000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:31.649 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] "is-owner": true, run task successfully found first match line: 457114316080152584 : [2025/04/04 15:45:32.130 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=158] [jobID=159] [query="CREATE TABLE IF NOT EXISTS `finishmark12000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114316080152584] [2025/04/04 15:45:32.136 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark12000] [2025/04/04 15:45:32.136 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114316080152584] [2025/04/04 15:45:32.141 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:32.230 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:32.236 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:32.242 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11144.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later owner pid: 4277 owner id e0f481c7-9017-420a-b47f-c5712e95ba9b [Fri Apr 4 15:45:33 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.44794481.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.table5 not exists for 2-th check, retry later table test.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 75-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x + tso='457114316245565441 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114316245565441 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:33 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c002d4d5-3433-4cf7-9c61-bc10ec0d543e {"id":"c002d4d5-3433-4cf7-9c61-bc10ec0d543e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752730} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d83af4 c002d4d5-3433-4cf7-9c61-bc10ec0d543e /tidb/cdc/default/default/upstream/7489360886006559165 {"id":7489360886006559165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c002d4d5-3433-4cf7-9c61-bc10ec0d543e {"id":"c002d4d5-3433-4cf7-9c61-bc10ec0d543e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752730} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d83af4 c002d4d5-3433-4cf7-9c61-bc10ec0d543e /tidb/cdc/default/default/upstream/7489360886006559165 {"id":7489360886006559165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c002d4d5-3433-4cf7-9c61-bc10ec0d543e {"id":"c002d4d5-3433-4cf7-9c61-bc10ec0d543e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752730} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d83af4 c002d4d5-3433-4cf7-9c61-bc10ec0d543e /tidb/cdc/default/default/upstream/7489360886006559165 {"id":7489360886006559165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 4b97ab93-08e7-48a0-97e1-0a8a1f1f7bce Info: {"upstream_id":7489360886006559165,"namespace":"default","id":"4b97ab93-08e7-48a0-97e1-0a8a1f1f7bce","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:45:33.385240311+08:00","start_ts":457114315326488577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114315326488577,"checkpoint_ts":457114315326488577,"checkpoint_time":"2025-04-04 15:45:28.754"} # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_nfs/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["consistent_replicate_nfs.usertable*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "457114313005465712" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:45:33.943 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:45:33.944 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:45:33.944 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:45:33.944 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:45:33.944 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114312448933929] [resolvedTs=457114313005465712] [2025/04/04 15:45:33.945 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=307.637ยตs] [2025/04/04 15:45:33.946 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:45:33.946 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:45:33.946 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] table common_1.v1 not exists for 1-th check, retry later [2025/04/04 15:45:33.951 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:45:33.955 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:45:33.957 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:45:33.957 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:45:33.957 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:45:33.957 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:45:33.962 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:45:33.964 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:45:33.964 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:45:33.965 +08:00] [INFO] [file.go:284] ["ignore logs which commitTs is greater than resolvedTs"] [filename=127.0.0.1:8300_a932b657-2572-4ba2-958a-886239ac6055_row_457114313031680104_36077687-71f3-445a-8332-51cc979b6629.log] [endTs=457114313005465712] [2025/04/04 15:45:33.967 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=22.119357ms] [2025/04/04 15:45:34.190 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}"] [checkpointTs=457114313005465712] [2025/04/04 15:45:34.190 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}"] [checkpointTs=457114313005465712] start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:34 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d1d710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316451872771 MinTableBarrierTs:457114316713754628 AdminJobType:noop} span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1d099b97-8024-47e7-93c9-d9ef945ca6e4 {"id":"1d099b97-8024-47e7-93c9-d9ef945ca6e4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b731a3 1d099b97-8024-47e7-93c9-d9ef945ca6e4 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316451872771,"min-table-barrier-ts":457114316713754628,"admin-job-type":0} /tidb/cdc/default/default/task/position/1d099b97-8024-47e7-93c9-d9ef945ca6e4/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d1d710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316451872771 MinTableBarrierTs:457114316713754628 AdminJobType:noop} span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1d099b97-8024-47e7-93c9-d9ef945ca6e4 {"id":"1d099b97-8024-47e7-93c9-d9ef945ca6e4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b731a3 1d099b97-8024-47e7-93c9-d9ef945ca6e4 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316451872771,"min-table-barrier-ts":457114316713754628,"admin-job-type":0} /tidb/cdc/default/default/task/position/1d099b97-8024-47e7-93c9-d9ef945ca6e4/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d1d710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316451872771 MinTableBarrierTs:457114316713754628 AdminJobType:noop} span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316713754627, checkpointTs: 457114316713754627, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1d099b97-8024-47e7-93c9-d9ef945ca6e4 {"id":"1d099b97-8024-47e7-93c9-d9ef945ca6e4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b731a3 1d099b97-8024-47e7-93c9-d9ef945ca6e4 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316451872771,"min-table-barrier-ts":457114316713754628,"admin-job-type":0} /tidb/cdc/default/default/task/position/1d099b97-8024-47e7-93c9-d9ef945ca6e4/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:45:34.291 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=2646] [appliedDDLCount=0] [currentCheckpoint=457114313005465712] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:45:34.291 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] Apply redo log successfully check diff failed 1-th time, retry later [Fri Apr 4 15:45:34 CST 2025] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1119511197.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:45:34.838 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark13000] [2025/04/04 15:45:34.838 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark13000] check diff successfully [Fri Apr 4 15:45:34 CST 2025] <<<<<< run test case consistent_replicate_nfs success! >>>>>> count(*) 47484 Starting Upstream TiDB... table test.table5 not exists for 3-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 5-th check, retry later table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later table common_1.v1 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 76-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:36 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00432a1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114317144981511 MinTableBarrierTs:457114317144981511 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ddb98e-18f5-4cb2-afea-04efccd29c41 {"id":"85ddb98e-18f5-4cb2-afea-04efccd29c41","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752733} /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460b07 85ddb98e-18f5-4cb2-afea-04efccd29c41 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114317144981511,"min-table-barrier-ts":457114317144981511,"admin-job-type":0} /tidb/cdc/default/default/task/position/85ddb98e-18f5-4cb2-afea-04efccd29c41/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00432a1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114317144981511 MinTableBarrierTs:457114317144981511 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ddb98e-18f5-4cb2-afea-04efccd29c41 {"id":"85ddb98e-18f5-4cb2-afea-04efccd29c41","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752733} /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460b07 85ddb98e-18f5-4cb2-afea-04efccd29c41 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114317144981511,"min-table-barrier-ts":457114317144981511,"admin-job-type":0} /tidb/cdc/default/default/task/position/85ddb98e-18f5-4cb2-afea-04efccd29c41/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00432a1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114317144981511 MinTableBarrierTs:457114317144981511 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ddb98e-18f5-4cb2-afea-04efccd29c41 {"id":"85ddb98e-18f5-4cb2-afea-04efccd29c41","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752733} /tidb/cdc/default/__cdc_meta__/capture/e0f481c7-9017-420a-b47f-c5712e95ba9b {"id":"e0f481c7-9017-420a-b47f-c5712e95ba9b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460a63 e0f481c7-9017-420a-b47f-c5712e95ba9b /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460b07 85ddb98e-18f5-4cb2-afea-04efccd29c41 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114317144981511,"min-table-barrier-ts":457114317144981511,"admin-job-type":0} /tidb/cdc/default/default/task/position/85ddb98e-18f5-4cb2-afea-04efccd29c41/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e0f481c7-9017-420a-b47f-c5712e95ba9b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "e0f481c7-9017-420a-b47f-c5712e95ba9b" | grep id Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.table5 exists table test.table6 not exists for 1-th check, retry later check diff successfully check_ts_not_forward ddl-only-block-related-table table test.finish_mark not exists for 6-th check, retry later table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later [2025/04/04 15:45:36.838 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [2025/04/04 15:45:36.845 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark13000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:36.847 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] [2025/04/04 15:45:33.918 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 12"] cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) found first match line: 457114317263994887 : [2025/04/04 15:45:37.129 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=160] [jobID=161] [query="CREATE TABLE IF NOT EXISTS `finishmark13000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114317263994887] [2025/04/04 15:45:37.731 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark13000] [2025/04/04 15:45:37.731 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114317263994887] [2025/04/04 15:45:37.733 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:37.737 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:37.742 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:37.747 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] "cluster-id": "default" "id": "85ddb98e-18f5-4cb2-afea-04efccd29c41", "cluster-id": "default" run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:37 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/463cd28e-1701-4a35-910f-dd1e04868007 {"id":"463cd28e-1701-4a35-910f-dd1e04868007","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752735} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2f3625b 463cd28e-1701-4a35-910f-dd1e04868007 /tidb/cdc/default/default/upstream/7489360913722737017 {"id":7489360913722737017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/463cd28e-1701-4a35-910f-dd1e04868007 {"id":"463cd28e-1701-4a35-910f-dd1e04868007","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752735} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2f3625b 463cd28e-1701-4a35-910f-dd1e04868007 /tidb/cdc/default/default/upstream/7489360913722737017 {"id":7489360913722737017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/463cd28e-1701-4a35-910f-dd1e04868007 {"id":"463cd28e-1701-4a35-910f-dd1e04868007","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752735} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2f3625b 463cd28e-1701-4a35-910f-dd1e04868007 /tidb/cdc/default/default/upstream/7489360913722737017 {"id":7489360913722737017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11249.out cli changefeed create --start-ts=457114316245565441 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name table ddl_manager.finish_mark exists check diff successfully [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360913722737017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:38.256603326+08:00","start_ts":457114316245565441,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114316245565441,"checkpoint_ts":457114316245565441,"checkpoint_time":"2025-04-04 15:45:32.260"} PASS table common_1.v1 not exists for 3-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2025/04/04 15:45:38.919 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 25"] Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.table6 exists check diff successfully check diff successfully table test.finish_mark not exists for 7-th check, retry later table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... Resume changefeed with checkpointTs1 457114313886531585 capture_id: 85ddb98e-18f5-4cb2-afea-04efccd29c41 wait process cdc.test exit for 3-th time... table test.table3 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360956266287042 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0cc8fc000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:6591, start at 2025-04-04 15:45:37.387033735 +0800 CST m=+2.201360229 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:37.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:37.394 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:37.394 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360956266287042 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0cc8fc000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:6591, start at 2025-04-04 15:45:37.387033735 +0800 CST m=+2.201360229 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:37.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:37.394 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:37.394 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360954379126822 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ccce4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:6657, start at 2025-04-04 15:45:37.639838042 +0800 CST m=+2.378839101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:37.648 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:37.643 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:37.643 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later wait process cdc.test exit for 4-th time... wait process cdc.test exit for 2-th time... table common_1.v1 exists table common_1.recover_and_insert not exists for 1-th check, retry later [2025/04/04 15:45:39.937 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark14000] [2025/04/04 15:45:39.937 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark14000] wait process cdc.test exit for 5-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:40 CST 2025] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later wait process cdc.test exit for 6-th time... start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Fri Apr 4 15:45:41 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> table test.finish_mark not exists for 8-th check, retry later cdc.test: no process found wait process cdc.test exit for 7-th time... process cdc.test already exit [Fri Apr 4 15:45:41 CST 2025] <<<<<< run test case ddl_manager success! >>>>>> table test.`simple-dash` not exists for 2-th check, retry later + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.79677969.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.table3 not exists for 2-th check, retry later table common_1.recover_and_insert not exists for 2-th check, retry later [2025/04/04 15:45:41.938 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2025/04/04 15:45:41.945 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark14000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:41.946 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later found first match line: 457114318574714884 : [2025/04/04 15:45:42.129 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=162] [jobID=163] [query="CREATE TABLE IF NOT EXISTS `finishmark14000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114318574714884] [2025/04/04 15:45:42.839 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark14000] [2025/04/04 15:45:42.840 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114318574714884] [2025/04/04 15:45:42.842 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:42.845 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:42.848 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:42.853 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360972912387044 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d011c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:10084, start at 2025-04-04 15:45:40.9419516 +0800 CST m=+2.272926727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:40.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:40.935 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:40.935 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360972912387044 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d011c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:10084, start at 2025-04-04 15:45:40.9419516 +0800 CST m=+2.272926727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:40.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:40.935 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:40.935 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360975070684727 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d03b00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-xh4jw-bv9zd-wkzjz, pid:10149, start at 2025-04-04 15:45:41.11230787 +0800 CST m=+2.369623926 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:41.119 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:41.100 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:41.100 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.table3 not exists for 3-th check, retry later table common_1.recover_and_insert not exists for 3-th check, retry later table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114319181578251, "checkpoint_time": "2025-04-04 15:45:43.460", "error": null }' + echo '{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114319181578251, "checkpoint_time": "2025-04-04 15:45:43.460", "error": null }' { "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114319181578251, "checkpoint_time": "2025-04-04 15:45:43.460", "error": null } ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114319181578251, '"checkpoint_time":' '"2025-04-04' '15:45:43.460",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114319181578251, '"checkpoint_time":' '"2025-04-04' '15:45:43.460",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 [2025/04/04 15:45:44.640 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark15000] [2025/04/04 15:45:44.640 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark15000] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:44 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee912a05-9d3e-4c77-8939-abf906cf783f {"id":"ee912a05-9d3e-4c77-8939-abf906cf783f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752741} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc311ced8 ee912a05-9d3e-4c77-8939-abf906cf783f /tidb/cdc/default/default/upstream/7489360956266287042 {"id":7489360956266287042,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee912a05-9d3e-4c77-8939-abf906cf783f {"id":"ee912a05-9d3e-4c77-8939-abf906cf783f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752741} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc311ced8 ee912a05-9d3e-4c77-8939-abf906cf783f /tidb/cdc/default/default/upstream/7489360956266287042 {"id":7489360956266287042,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee912a05-9d3e-4c77-8939-abf906cf783f {"id":"ee912a05-9d3e-4c77-8939-abf906cf783f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752741} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc311ced8 ee912a05-9d3e-4c77-8939-abf906cf783f /tidb/cdc/default/default/upstream/7489360956266287042 {"id":7489360956266287042,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.8023.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_add_table/conf/changefeed.toml Create changefeed successfully! ID: fff99001-b86c-4c52-a912-0cf6a0a43f99 Info: {"upstream_id":7489360956266287042,"namespace":"default","id":"fff99001-b86c-4c52-a912-0cf6a0a43f99","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:45.094624316+08:00","start_ts":457114319583707143,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!test.t_1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114319583707143,"checkpoint_ts":457114319583707143,"checkpoint_time":"2025-04-04 15:45:44.994"} PASS + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11396.out cli tso query --pd=http://127.0.0.1:2379 coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 10-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11513.out cli changefeed --changefeed-id custom-changefeed-name pause Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table test.table3 exists table test.table4 not exists for 1-th check, retry later PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table common_1.recover_and_insert exists table common_1.finish_mark not exists for 1-th check, retry later [2025/04/04 15:45:46.640 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] + set +x [2025/04/04 15:45:46.733 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark15000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:46.734 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.t_1 not exists for 1-th check, retry later + set +x + tso='457114319699574786 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114319699574786 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:45:46 CST 2025] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1143411436.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x table test.finish_mark not exists for 11-th check, retry later table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later table test.table4 exists found first match line: 457114319911649293 : [2025/04/04 15:45:47.229 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=164] [jobID=165] [query="CREATE TABLE IF NOT EXISTS `finishmark15000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114319911649293] [2025/04/04 15:45:47.841 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark15000] [2025/04/04 15:45:47.841 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114319911649293] [2025/04/04 15:45:47.844 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:47.847 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:47.851 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:47.856 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table common_1.finish_mark not exists for 2-th check, retry later empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.finish_mark not exists for 12-th check, retry later [2025/04/04 15:45:49.138 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark16000] [2025/04/04 15:45:49.138 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark16000] table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later table test.t_1 exists table test.t_2 exists table test.t_3 exists table test.t_4 exists table test.t_5 exists table test.t_6 exists table test.t_7 exists table test.t_8 exists table test.t_9 exists table test.t_10 exists table test.t_11 exists table test.t_12 exists table test.t_13 exists table test.t_14 exists table test.t_15 exists table test.t_16 exists table test.t_17 exists table test.t_18 exists table test.t_19 exists table test.t_20 exists table test.t_21 exists table test.t_22 exists table test.t_23 exists table test.t_24 exists table test.t_25 exists table test.t_26 exists table test.t_27 exists table test.t_28 exists table test.t_29 exists table test.t_30 exists table test.t_31 exists table test.t_32 exists table test.t_33 exists table test.t_34 exists table test.t_35 exists table test.t_36 exists table test.t_37 exists table test.t_38 exists table test.t_39 exists table test.t_40 exists table test.t_41 exists table test.t_42 exists table test.t_43 exists table test.t_44 exists table test.t_45 exists table test.t_46 exists table test.t_47 exists table test.t_48 exists table test.t_49 exists table test.t_50 exists table test.t_51 exists table test.t_52 exists table test.t_53 exists table test.t_54 exists table test.t_55 exists table test.t_56 exists table test.t_57 exists table test.t_58 exists table test.t_59 exists table test.t_60 exists table test.t_61 exists table test.t_62 exists table test.t_63 exists table test.t_64 exists cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:49 CST 2025] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.85108512.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t_65 exists table test.t_66 exists table test.t_67 exists table test.t_68 exists table test.t_69 exists table test.t_70 exists table test.t_71 exists table test.t_72 exists table test.t_73 exists table test.t_74 exists table test.t_75 exists table test.t_76 exists table test.t_77 exists table test.t_78 exists table test.t_79 exists table test.t_80 exists table test.t_81 exists table test.t_82 exists table test.t_83 exists table test.t_84 exists table test.t_85 exists table test.t_86 exists table test.t_87 exists table test.t_88 exists table test.t_89 exists table test.t_90 exists table test.t_91 exists table test.t_92 exists table test.t_93 exists table test.t_94 exists table test.t_95 exists table test.t_96 exists table test.t_97 exists table test.t_98 exists table test.t_99 exists table test.t_100 exists + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95debc5f-a216-470f-abc1-4a4976db5682 {"id":"95debc5f-a216-470f-abc1-4a4976db5682","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752747} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3254cd2 95debc5f-a216-470f-abc1-4a4976db5682 /tidb/cdc/default/default/upstream/7489360972912387044 {"id":7489360972912387044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95debc5f-a216-470f-abc1-4a4976db5682 {"id":"95debc5f-a216-470f-abc1-4a4976db5682","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752747} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3254cd2 95debc5f-a216-470f-abc1-4a4976db5682 /tidb/cdc/default/default/upstream/7489360972912387044 {"id":7489360972912387044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95debc5f-a216-470f-abc1-4a4976db5682 {"id":"95debc5f-a216-470f-abc1-4a4976db5682","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752747} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3254cd2 95debc5f-a216-470f-abc1-4a4976db5682 /tidb/cdc/default/default/upstream/7489360972912387044 {"id":7489360972912387044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11488.out cli changefeed create --start-ts=457114319699574786 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table common_1.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360999566627994 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d77a0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:4099, start at 2025-04-04 15:45:48.556133652 +0800 CST m=+2.791958177 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:48.563 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:48.569 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:48.569 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360999566627994 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d77a0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:4099, start at 2025-04-04 15:45:48.556133652 +0800 CST m=+2.791958177 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:48.563 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:48.569 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:48.569 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361005401368427 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d7a040006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9cfmv-g5xz7-09qt2, pid:4157, start at 2025-04-04 15:45:48.682808522 +0800 CST m=+2.836015397 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:48.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:48.723 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:48.723 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t_100 not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114319705866247, "checkpoint_time": "2025-04-04 15:45:45.460", "error": null }' + echo '{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114319705866247, "checkpoint_time": "2025-04-04 15:45:45.460", "error": null }' { "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114319705866247, "checkpoint_time": "2025-04-04 15:45:45.460", "error": null } ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114319705866247, '"checkpoint_time":' '"2025-04-04' '15:45:45.460",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114319705866247, '"checkpoint_time":' '"2025-04-04' '15:45:45.460",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11590.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Create changefeed successfully! ID: fa75146e-8dbf-4d42-9688-7191cb7cc399 Info: {"upstream_id":7489360972912387044,"namespace":"default","id":"fa75146e-8dbf-4d42-9688-7191cb7cc399","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:50.407690433+08:00","start_ts":457114319699574786,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114319699574786,"checkpoint_ts":457114319699574786,"checkpoint_time":"2025-04-04 15:45:45.436"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360642715955812 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0849240007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:8925, start at 2025-04-04 15:44:23.637148323 +0800 CST m=+2.403188352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.625 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.625 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360639630816058 TiDB Cluster ID. tikv_gc_leader_uuid 657ff084b48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-zl944-bffst-ncbbk, pid:9001, start at 2025-04-04 15:44:23.78426212 +0800 CST m=+2.439780038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:23.825 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:23.812 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:23.812 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14148.out cli tso query --pd=http://127.0.0.1:2379 nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360913722737017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:38.256603326+08:00","start_ts":457114316245565441,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114319705866247,"checkpoint_time":"2025-04-04 15:45:45.460"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:45:51.138 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] [2025/04/04 15:45:51.145 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark16000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:51.147 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: mysql... <<================= table test.finish_mark not exists for 13-th check, retry later + set +x found first match line: 457114320877387784 : [2025/04/04 15:45:50.230 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=166] [jobID=167] [query="CREATE TABLE IF NOT EXISTS `finishmark16000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114320877387784] [2025/04/04 15:45:52.047 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark16000] [2025/04/04 15:45:52.047 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114320877387784] [2025/04/04 15:45:52.050 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:52.054 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table resourcecontrol.finish_mark not exists for 1-th check, retry later table common_1.finish_mark not exists for 4-th check, retry later [2025/04/04 15:45:52.133 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:52.139 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + set +x Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x + tso='457114321093394433 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114321093394433 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114321093394433 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:45:52 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1418614188.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11632.out cli changefeed --changefeed-id custom-changefeed-name resume + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:52 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3376 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebbf2e5-1591-4230-9487-e8c8e9c538b3 {"id":"4ebbf2e5-1591-4230-9487-e8c8e9c538b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752749} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c13685 4ebbf2e5-1591-4230-9487-e8c8e9c538b3 /tidb/cdc/default/default/changefeed/info/a0371b7d-6f18-4208-ba42-f7549915de25 {"upstream-id":7489360863193495274,"namespace":"default","changefeed-id":"a0371b7d-6f18-4208-ba42-f7549915de25","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:45:23.159646495+08:00","start-ts":457114313886531585,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:45:49.865744206+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114320859561986} /tidb/cdc/default/default/changefeed/status/a0371b7d-6f18-4208-ba42-f7549915de25 {"checkpoint-ts":457114315734384661,"min-table-barrier-ts":457114315734384661,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360863193495274 {"id":7489360863193495274,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebbf2e5-1591-4230-9487-e8c8e9c538b3 {"id":"4ebbf2e5-1591-4230-9487-e8c8e9c538b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752749} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c13685 4ebbf2e5-1591-4230-9487-e8c8e9c538b3 /tidb/cdc/default/default/changefeed/info/a0371b7d-6f18-4208-ba42-f7549915de25 {"upstream-id":7489360863193495274,"namespace":"default","changefeed-id":"a0371b7d-6f18-4208-ba42-f7549915de25","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:45:23.159646495+08:00","start-ts":457114313886531585,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:45:49.865744206+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114320859561986} /tidb/cdc/default/default/changefeed/status/a0371b7d-6f18-4208-ba42-f7549915de25 {"checkpoint-ts":457114315734384661,"min-table-barrier-ts":457114315734384661,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360863193495274 + grep -q 'failed to get info:' {"id":7489360863193495274,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebbf2e5-1591-4230-9487-e8c8e9c538b3 {"id":"4ebbf2e5-1591-4230-9487-e8c8e9c538b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752749} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2c13685 4ebbf2e5-1591-4230-9487-e8c8e9c538b3 /tidb/cdc/default/default/changefeed/info/a0371b7d-6f18-4208-ba42-f7549915de25 {"upstream-id":7489360863193495274,"namespace":"default","changefeed-id":"a0371b7d-6f18-4208-ba42-f7549915de25","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:45:23.159646495+08:00","start-ts":457114313886531585,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:45:49.865744206+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114320859561986} /tidb/cdc/default/default/changefeed/status/a0371b7d-6f18-4208-ba42-f7549915de25 {"checkpoint-ts":457114315734384661,"min-table-barrier-ts":457114315734384661,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360863193495274 {"id":7489360863193495274,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_changefeed_state http://127.0.0.1:2379 a0371b7d-6f18-4208-ba42-f7549915de25 failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=a0371b7d-6f18-4208-ba42-f7549915de25 + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c a0371b7d-6f18-4208-ba42-f7549915de25 -s + info='{ "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "failed", "checkpoint_tso": 457114315734384661, "checkpoint_time": "2025-04-04 15:45:30.310", "error": { "time": "2025-04-04T15:45:49.865744206+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "failed", "checkpoint_tso": 457114315734384661, "checkpoint_time": "2025-04-04 15:45:30.310", "error": { "time": "2025-04-04T15:45:49.865744206+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "failed", "checkpoint_tso": 457114315734384661, "checkpoint_time": "2025-04-04 15:45:30.310", "error": { "time": "2025-04-04T15:45:49.865744206+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489360863193495274, '"namespace":' '"default",' '"id":' '"a0371b7d-6f18-4208-ba42-f7549915de25",' '"state":' '"failed",' '"checkpoint_tso":' 457114315734384661, '"checkpoint_time":' '"2025-04-04' '15:45:30.310",' '"error":' '{' '"time":' '"2025-04-04T15:45:49.865744206+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360863193495274, '"namespace":' '"default",' '"id":' '"a0371b7d-6f18-4208-ba42-f7549915de25",' '"state":' '"failed",' '"checkpoint_tso":' 457114315734384661, '"checkpoint_time":' '"2025-04-04' '15:45:30.310",' '"error":' '{' '"time":' '"2025-04-04T15:45:49.865744206+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully \033[0;36m<<< Run all test success >>>\033[0m [Fri Apr 4 15:45:52 CST 2025] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.54245426.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) table test.t_100 not exists for 2-th check, retry later [Pipeline] // cache table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 a0371b7d-6f18-4208-ba42-f7549915de25 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=a0371b7d-6f18-4208-ba42-f7549915de25 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c a0371b7d-6f18-4208-ba42-f7549915de25 -s [Pipeline] } nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... [Pipeline] // dir + info='{ "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "normal", "checkpoint_tso": 457114321632886791, "checkpoint_time": "2025-04-04 15:45:52.811", "error": null }' + echo '{ "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "normal", "checkpoint_tso": 457114321632886791, "checkpoint_time": "2025-04-04 15:45:52.811", "error": null }' { "upstream_id": 7489360863193495274, "namespace": "default", "id": "a0371b7d-6f18-4208-ba42-f7549915de25", "state": "normal", "checkpoint_tso": 457114321632886791, "checkpoint_time": "2025-04-04 15:45:52.811", "error": null } ++ echo '{' '"upstream_id":' 7489360863193495274, '"namespace":' '"default",' '"id":' '"a0371b7d-6f18-4208-ba42-f7549915de25",' '"state":' '"normal",' '"checkpoint_tso":' 457114321632886791, '"checkpoint_time":' '"2025-04-04' '15:45:52.811",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360863193495274, '"namespace":' '"default",' '"id":' '"a0371b7d-6f18-4208-ba42-f7549915de25",' '"state":' '"normal",' '"checkpoint_tso":' 457114321632886791, '"checkpoint_time":' '"2025-04-04' '15:45:52.811",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully [Pipeline] } [Pipeline] // withCredentials [Pipeline] } table test.finish_mark not exists for 14-th check, retry later [Pipeline] // timeout table test1.table1 not exists for 1-th check, retry later [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table resourcecontrol.finish_mark exists [Pipeline] // node table common_1.finish_mark exists [Pipeline] } [Pipeline] // podTemplate check diff successfully [Pipeline] } check diff successfully [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x [2025/04/04 15:45:54.632 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark17000] [2025/04/04 15:45:54.632 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark17000] wait process cdc.test exit for 1-th time... table test.t_100 not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:55 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 {"id":"7c8e37a0-8654-4f7a-bf1a-b1e753e07b54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33f04d8 7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 /tidb/cdc/default/default/upstream/7489360999566627994 {"id":7489360999566627994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 {"id":"7c8e37a0-8654-4f7a-bf1a-b1e753e07b54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33f04d8 7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 /tidb/cdc/default/default/upstream/7489360999566627994 {"id":7489360999566627994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 {"id":"7c8e37a0-8654-4f7a-bf1a-b1e753e07b54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33f04d8 7c8e37a0-8654-4f7a-bf1a-b1e753e07b54 /tidb/cdc/default/default/upstream/7489360999566627994 {"id":7489360999566627994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 15-th check, retry later wait process cdc.test exit for 2-th time... table test1.table1 not exists for 2-th check, retry later table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later Create changefeed successfully! ID: fa259769-c825-450a-b433-b66362c24787 Info: {"upstream_id":7489360999566627994,"namespace":"default","id":"fa259769-c825-450a-b433-b66362c24787","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:45:55.613426905+08:00","start_ts":457114321491066881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114321491066881,"checkpoint_ts":457114321491066881,"checkpoint_time":"2025-04-04 15:45:52.270"} cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:55 CST 2025] <<<<<< run test case resourcecontrol success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:55 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360642715955812 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:44:33.050645127 +0800 CST StartTs:457114299417231361 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042321b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114300714844166} {CheckpointTs:457114304188514309 MinTableBarrierTs:457114304188514309 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/capture/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd {"id":"5b0e514f-08de-41b7-a5d7-5b9ce833a0bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33c36d9 5b0e514f-08de-41b7-a5d7-5b9ce833a0bd /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360642715955812,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:44:33.050645127+08:00","start-ts":457114299417231361,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114300714844166} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114304188514309,"min-table-barrier-ts":457114304188514309,"admin-job-type":0} /tidb/cdc/default/default/task/position/24d62d64-4b48-4c86-b429-2e940c301d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360642715955812 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:44:33.050645127 +0800 CST StartTs:457114299417231361 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042321b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114300714844166} {CheckpointTs:457114304188514309 MinTableBarrierTs:457114304188514309 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/capture/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd {"id":"5b0e514f-08de-41b7-a5d7-5b9ce833a0bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33c36d9 5b0e514f-08de-41b7-a5d7-5b9ce833a0bd /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360642715955812,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:44:33.050645127+08:00","start-ts":457114299417231361,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114300714844166} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114304188514309,"min-table-barrier-ts":457114304188514309,"admin-job-type":0} /tidb/cdc/default/default/task/position/24d62d64-4b48-4c86-b429-2e940c301d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360642715955812 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:44:33.050645127 +0800 CST StartTs:457114299417231361 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042321b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114300714844166} {CheckpointTs:457114304188514309 MinTableBarrierTs:457114304188514309 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/24d62d64-4b48-4c86-b429-2e940c301d93 {"id":"24d62d64-4b48-4c86-b429-2e940c301d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752669} /tidb/cdc/default/__cdc_meta__/capture/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd {"id":"5b0e514f-08de-41b7-a5d7-5b9ce833a0bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752752} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1fac9e5 24d62d64-4b48-4c86-b429-2e940c301d93 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33c36d9 5b0e514f-08de-41b7-a5d7-5b9ce833a0bd /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360642715955812,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:44:33.050645127+08:00","start-ts":457114299417231361,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114300714844166} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114304188514309,"min-table-barrier-ts":457114304188514309,"admin-job-type":0} /tidb/cdc/default/default/task/position/24d62d64-4b48-4c86-b429-2e940c301d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/5b0e514f-08de-41b7-a5d7-5b9ce833a0bd/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360642715955812 {"id":7489360642715955812,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114321093394433 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14244.out cli changefeed create --start-ts=457114321093394433 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:55 CST 2025] <<<<<< run test case common_1 success! >>>>>> Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489360642715955812,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:45:56.504759958+08:00","start_ts":457114321093394433,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114321093394433,"checkpoint_ts":457114321093394433,"checkpoint_time":"2025-04-04 15:45:50.753"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:45:56.632 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] [2025/04/04 15:45:56.641 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark17000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:45:56.643 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.t_100 not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark not exists for 16-th check, retry later table processor_delay.t36 exists table processor_delay.t37 not exists for 1-th check, retry later found first match line: 457114322292965389 : [2025/04/04 15:45:56.157 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=168] [jobID=169] [query="CREATE TABLE IF NOT EXISTS `finishmark17000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114322292965389] [2025/04/04 15:45:57.339 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark17000] [2025/04/04 15:45:57.339 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114322292965389] [2025/04/04 15:45:57.341 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:57.345 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:45:57.351 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:45:57.357 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114322602557442, "checkpoint_time": "2025-04-04 15:45:56.510", "error": null }' + echo '{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114322602557442, "checkpoint_time": "2025-04-04 15:45:56.510", "error": null }' { "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114322602557442, "checkpoint_time": "2025-04-04 15:45:56.510", "error": null } ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114322602557442, '"checkpoint_time":' '"2025-04-04' '15:45:56.510",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114322602557442, '"checkpoint_time":' '"2025-04-04' '15:45:56.510",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11704.out cli changefeed --changefeed-id custom-changefeed-name remove table test1.table1 exists table test1.table2 not exists for 1-th check, retry later + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114322864701442 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.t_100 not exists for 5-th check, retry later coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark not exists for 17-th check, retry later table test1.table2 exists check diff successfully + set +x [2025/04/04 15:45:59.731 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark18000] [2025/04/04 15:45:59.731 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark18000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.t_100 not exists for 6-th check, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later empty select id, val from test.availability3 where id=1 run task successfully test_gap_between_watch_capture pass wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 18-th check, retry later wait process cdc.test exit for 2-th time... [2025/04/04 15:46:01.732 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [2025/04/04 15:46:01.739 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark18000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:46:01.741 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:02 CST 2025] <<<<<< run test case changefeed_resume_with_checkpoint_ts success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case delete_owner_key [Fri Apr 4 15:46:02 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.48224824.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 found first match line: 457114323870023682 : [2025/04/04 15:46:02.330 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=170] [jobID=171] [query="CREATE TABLE IF NOT EXISTS `finishmark18000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114323870023682] [2025/04/04 15:46:02.953 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark18000] [2025/04/04 15:46:02.953 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114323870023682] table test.t_100 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t39 exists [2025/04/04 15:46:03.030 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:03.034 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:46:03.038 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:03.043 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11774.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name table processor_delay.t40 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360913722737017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:46:03.453960952+08:00","start_ts":457114324398243844,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114324398243844,"checkpoint_ts":457114324398243844,"checkpoint_time":"2025-04-04 15:46:03.360"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 19-th check, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later [2025/04/04 15:46:04.131 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark19000] [2025/04/04 15:46:04.131 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark19000] table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later + set +x table test.t_100 not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 20-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } table testSync.usertable exists table testSync.simple1 exists table testSync.simple2 exists [Pipeline] // withCredentials \033[0;36m<<< Run all test success >>>\033[0m + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:05 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00430f9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114323698319368 MinTableBarrierTs:457114324760002561 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114323698319368,"min-table-barrier-ts":457114324760002561,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00430f9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114323698319368 MinTableBarrierTs:457114324760002561 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114323698319368,"min-table-barrier-ts":457114324760002561,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00430f9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114323698319368 MinTableBarrierTs:457114324760002561 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114323698319368, checkpointTs: 457114323698319368, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114323698319368,"min-table-barrier-ts":457114324760002561,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // stage [Pipeline] // cache [Pipeline] } [Pipeline] } [Pipeline] // container [Pipeline] // dir table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later [Pipeline] } [Pipeline] } [2025/04/04 15:46:06.131 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] [2025/04/04 15:46:06.142 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark19000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:46:06.146 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] [Pipeline] // withEnv [Pipeline] // withCredentials [Pipeline] } [Pipeline] } table test.t_100 not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // node [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] // podTemplate [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } "is-owner": true, run task successfully table test.finish_mark not exists for 21-th check, retry later [Pipeline] // withEnv [Pipeline] // container found first match line: 457114324944814085 : [2025/04/04 15:46:06.330 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=172] [jobID=173] [query="CREATE TABLE IF NOT EXISTS `finishmark19000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114324944814085] [2025/04/04 15:46:07.344 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark19000] [2025/04/04 15:46:07.344 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114324944814085] [2025/04/04 15:46:07.347 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:07.350 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:46:07.354 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:07.359 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // withEnv [Pipeline] } [Pipeline] } table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114325486141443, "checkpoint_time": "2025-04-04 15:46:07.510", "error": null }' + echo '{ "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114325486141443, "checkpoint_time": "2025-04-04 15:46:07.510", "error": null }' { "upstream_id": 7489360913722737017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114325486141443, "checkpoint_time": "2025-04-04 15:46:07.510", "error": null } ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114325486141443, '"checkpoint_time":' '"2025-04-04' '15:46:07.510",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360913722737017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114325486141443, '"checkpoint_time":' '"2025-04-04' '15:46:07.510",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } owner pid: 4827 owner id 9a4133e4-9e66-4b71-8abc-2fff71343467 default owner key /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 [Fri Apr 4 15:46:08 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.49434945.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t_100 not exists for 10-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11877.out cli unsafe delete-service-gc-safepoint ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361074549424038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e9b60000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:6838, start at 2025-04-04 15:46:07.206174329 +0800 CST m=+4.139935104 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:07.214 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:07.192 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:07.192 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:46:08.938 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20001] [2025/04/04 15:46:08.938 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:46:08.938 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20002] [2025/04/04 15:46:08.938 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:46:08.939 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20003] [2025/04/04 15:46:08.939 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:46:08.940 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20004] [2025/04/04 15:46:08.940 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:46:09.443 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark20000] [2025/04/04 15:46:09.443 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20004] [2025/04/04 15:46:09.443 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark20000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361074549424038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e9b60000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:6838, start at 2025-04-04 15:46:07.206174329 +0800 CST m=+4.139935104 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:07.214 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:07.192 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:07.192 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361074391697396 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e9b24000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:6909, start at 2025-04-04 15:46:07.220793636 +0800 CST m=+4.053112381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:07.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:07.227 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:07.227 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 22-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11917.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 table test.t_100 not exists for 11-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.8135.out cli tso query --pd=http://127.0.0.1:2379 table test.finish_mark not exists for 23-th check, retry later [2025/04/04 15:46:11.444 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] [2025/04/04 15:46:11.450 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark20000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:46:11.451 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114326072033288 : [2025/04/04 15:46:10.257 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=174] [jobID=175] [query="CREATE TABLE IF NOT EXISTS `finishmark20000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114326072033288] [2025/04/04 15:46:11.649 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark20000] [2025/04/04 15:46:11.649 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114326072033288] [2025/04/04 15:46:11.651 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:11.655 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:46:11.658 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:46:11.662 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:46:11.662 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:46:11.662 +08:00] [INFO] [case.go:620] ["bank test finished"] [valid=20] [tries=20] [ratio=1] + cleanup_process cdc.test + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:11 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031ad3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114326332866564 MinTableBarrierTs:457114326332866564 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2454898d-1850-49c0-92fb-80576b7c1ecd {"id":"2454898d-1850-49c0-92fb-80576b7c1ecd","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c93 2454898d-1850-49c0-92fb-80576b7c1ecd /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114326332866564,"min-table-barrier-ts":457114326332866564,"admin-job-type":0} /tidb/cdc/default/default/task/position/2454898d-1850-49c0-92fb-80576b7c1ecd/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031ad3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114326332866564 MinTableBarrierTs:457114326332866564 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2454898d-1850-49c0-92fb-80576b7c1ecd {"id":"2454898d-1850-49c0-92fb-80576b7c1ecd","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c93 2454898d-1850-49c0-92fb-80576b7c1ecd /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114326332866564,"min-table-barrier-ts":457114326332866564,"admin-job-type":0} /tidb/cdc/default/default/task/position/2454898d-1850-49c0-92fb-80576b7c1ecd/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031ad3b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114326332866564 MinTableBarrierTs:457114326332866564 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2454898d-1850-49c0-92fb-80576b7c1ecd {"id":"2454898d-1850-49c0-92fb-80576b7c1ecd","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/capture/9a4133e4-9e66-4b71-8abc-2fff71343467 {"id":"9a4133e4-9e66-4b71-8abc-2fff71343467","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752762} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c42 9a4133e4-9e66-4b71-8abc-2fff71343467 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460c93 2454898d-1850-49c0-92fb-80576b7c1ecd /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114326332866564,"min-table-barrier-ts":457114326332866564,"admin-job-type":0} /tidb/cdc/default/default/task/position/2454898d-1850-49c0-92fb-80576b7c1ecd/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9a4133e4-9e66-4b71-8abc-2fff71343467/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "9a4133e4-9e66-4b71-8abc-2fff71343467 default" | grep id table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark not exists for 3-th check, retry later table test.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:07 CST 2025] <<<<<< run test case many_pk_or_uk success! >>>>>> + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 [Pipeline] // cache [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } table test.t_100 exists [Pipeline] // withEnv ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } "id": "2454898d-1850-49c0-92fb-80576b7c1ecd", run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 4 15:46:13 CST 2025] <<<<<< run test case bank success! >>>>>>' [Fri Apr 4 15:46:13 CST 2025] <<<<<< run test case bank success! >>>>>> + stop_tidb_cluster run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later [Pipeline] } + set +x + tso='457114326648225793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114326648225793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:46:13 CST 2025] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.81748176.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 24-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 capture_id: 2454898d-1850-49c0-92fb-80576b7c1ecd 1 ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '2454898d-1850-49c0-92fb-80576b7c1ecd' 2454898d-1850-49c0-92fb-80576b7c1ecd run task successfully curl -X GET http://127.0.0.1:8300/status | grep '"is_owner": false' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 91500 0 --:--:-- --:--:-- --:--:-- 91500 "is_owner": false run task successfully run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12009.out cli unsafe resolve-lock --region=98 table test.finish_mark not exists for 25-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:16 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8cd6b39-1106-4f59-a3d0-d398421bbb69 {"id":"d8cd6b39-1106-4f59-a3d0-d398421bbb69","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752773} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc38474ec d8cd6b39-1106-4f59-a3d0-d398421bbb69 /tidb/cdc/default/default/upstream/7489361074549424038 {"id":7489361074549424038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8cd6b39-1106-4f59-a3d0-d398421bbb69 {"id":"d8cd6b39-1106-4f59-a3d0-d398421bbb69","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752773} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc38474ec d8cd6b39-1106-4f59-a3d0-d398421bbb69 /tidb/cdc/default/default/upstream/7489361074549424038 {"id":7489361074549424038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8cd6b39-1106-4f59-a3d0-d398421bbb69 {"id":"d8cd6b39-1106-4f59-a3d0-d398421bbb69","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752773} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc38474ec d8cd6b39-1106-4f59-a3d0-d398421bbb69 /tidb/cdc/default/default/upstream/7489361074549424038 {"id":7489361074549424038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.8228.out cli changefeed create --start-ts=457114326648225793 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12044.out cli unsafe resolve-lock --region=98 --ts=457114326731325446 Create changefeed successfully! ID: 09ba2fe2-4929-4109-af35-ec98f1a3e4b0 Info: {"upstream_id":7489361074549424038,"namespace":"default","id":"09ba2fe2-4929-4109-af35-ec98f1a3e4b0","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:46:16.948373741+08:00","start_ts":457114326648225793,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114326648225793,"checkpoint_ts":457114326648225793,"checkpoint_time":"2025-04-04 15:46:11.943"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 26-th check, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later + set +x go: downloading go1.23.4 (linux/amd64) run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361033720874216 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0f4fac000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:9784, start at 2025-04-04 15:46:18.752410238 +0800 CST m=+24.159982700 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:18.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:18.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:18.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361033720874216 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0f4fac000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:9784, start at 2025-04-04 15:46:18.752410238 +0800 CST m=+24.159982700 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:18.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:18.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:18.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361040311120562 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0df1a4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:9865, start at 2025-04-04 15:45:56.377713428 +0800 CST m=+1.709237025 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:56.402 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:56.378 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:56.378 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 207k 6000 --:--:-- --:--:-- --:--:-- 212k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 27-th check, retry later wait process cdc.test exit for 3-th time... start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:46:20 CST 2025] <<<<<< run test case force_replicate_table success! >>>>>> [Fri Apr 4 15:46:21 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1115611158.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later table test.finish_mark not exists for 28-th check, retry later Verifying downstream PD is started... } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 178k 0 --:--:-- --:--:-- --:--:-- 178k { "version": "v9.0.0-beta.1-10-ge8bfbe4ba", "git_hash": "e8bfbe4ba98e05140ab733ee3d08dd3b99918721", "id": "ac5a87b3-78db-4009-8a45-ac1bbe46e934", "pid": 11200, "is_owner": true }wait process cdc.test exit for 1-th time... table batch_add_table.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kill_owner_with_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:23 CST 2025] <<<<<< run test case cli_with_auth success! >>>>>> table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later table test.finish_mark not exists for 29-th check, retry later pass check, checkpoint tso not forward after 10s run task successfully table batch_add_table.finish_mark not exists for 2-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 wait process 7977 exit for 1-th time... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process 7977 exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:24 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba8334e1-e1d8-4270-acbc-7f9e87422221 {"id":"ba8334e1-e1d8-4270-acbc-7f9e87422221","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752781} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8835 ba8334e1-e1d8-4270-acbc-7f9e87422221 /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba8334e1-e1d8-4270-acbc-7f9e87422221 {"id":"ba8334e1-e1d8-4270-acbc-7f9e87422221","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752781} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8835 ba8334e1-e1d8-4270-acbc-7f9e87422221 /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba8334e1-e1d8-4270-acbc-7f9e87422221 {"id":"ba8334e1-e1d8-4270-acbc-7f9e87422221","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752781} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8835 ba8334e1-e1d8-4270-acbc-7f9e87422221 /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "readallfields"="true" "fieldcount"="100" "mysql.user"="root" "mysql.db"="consistent_replicate_storage_file" "mysql.port"="4000" "dotransactions"="false" "updateproportion"="0" "requestdistribution"="uniform" "scanproportion"="0" "recordcount"="5000" "insertproportion"="0" "mysql.host"="127.0.0.1" "workload"="core" "readproportion"="0" "operationcount"="0" "threadcount"="10" ********************************************** wait process 7977 exit for 3-th time... [2025/04/04 15:46:17.625 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 2 another is 5"] table processor_delay.t50 exists table test.finish_mark not exists for 30-th check, retry later wait process 7977 exit for 4-th time... nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/docker/go-units v0.5.0 empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/google/btree v1.1.2 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7977) - No such process wait process 7977 exit for 5-th time... process 7977 already exit [Fri Apr 4 15:46:26 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.85538555.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading google.golang.org/protobuf v1.35.2 go: downloading golang.org/x/tools v0.28.0 go: downloading golang.org/x/sys v0.28.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/carlmjohnson/flagext v0.21.0 table batch_add_table.finish_mark not exists for 3-th check, retry later check diff failed 1-th time, retry later go: downloading golang.org/x/time v0.7.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 Run finished, takes 1.903270467s INSERT - Takes(s): 1.9, Count: 5000, OPS: 2650.8, Avg(us): 3425, Min(us): 1196, Max(us): 67934, 95th(us): 5000, 99th(us): 47000 table consistent_replicate_storage_file.usertable not exists for 1-th check, retry later go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 start tidb cluster in /tmp/tidb_cdc_test/kill_owner_with_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 table test.finish_mark not exists for 31-th check, retry later go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/robfig/cron v1.2.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 table batch_add_table.finish_mark not exists for 4-th check, retry later go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully go: downloading github.com/klauspost/cpuid/v2 v2.2.4 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table consistent_replicate_storage_file.usertable exists table consistent_replicate_storage_file.check1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:29 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cfbe60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316779290664 MinTableBarrierTs:457114330882899975 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 {"id":"f00872e2-c7f2-4d7c-ba32-d4d2b8b13667","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752786} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b73381 f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316779290664,"min-table-barrier-ts":457114330882899975,"admin-job-type":0} /tidb/cdc/default/default/task/position/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cfbe60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316779290664 MinTableBarrierTs:457114330882899975 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 {"id":"f00872e2-c7f2-4d7c-ba32-d4d2b8b13667","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752786} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b73381 f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316779290664,"min-table-barrier-ts":457114330882899975,"admin-job-type":0} /tidb/cdc/default/default/task/position/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360848791361006 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:19.922664266 +0800 CST StartTs:457114312991571970 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cfbe60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114313004417030} {CheckpointTs:457114316779290664 MinTableBarrierTs:457114330882899975 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114316779290664, checkpointTs: 457114316779290664, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 {"id":"f00872e2-c7f2-4d7c-ba32-d4d2b8b13667","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752786} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2b73381 f00872e2-c7f2-4d7c-ba32-d4d2b8b13667 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360848791361006,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:45:19.922664266+08:00","start-ts":457114312991571970,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114313004417030} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114316779290664,"min-table-barrier-ts":457114330882899975,"admin-job-type":0} /tidb/cdc/default/default/task/position/f00872e2-c7f2-4d7c-ba32-d4d2b8b13667/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360848791361006 {"id":7489360848791361006,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully check_ts_forward ddl-only-block-related-table wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 32-th check, retry later wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:30 CST 2025] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> go: downloading github.com/pierrec/lz4 v2.6.1+incompatible nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361165820662906 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0fe608000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:19104, start at 2025-04-04 15:46:28.397946334 +0800 CST m=+2.171901183 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:28.405 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:28.404 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:28.404 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361165820662906 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0fe608000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:19104, start at 2025-04-04 15:46:28.397946334 +0800 CST m=+2.171901183 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:28.405 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:28.404 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:28.404 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361176210934060 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0feb74000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:19182, start at 2025-04-04 15:46:28.737435266 +0800 CST m=+2.432937410 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:28.757 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:28.753 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:28.753 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table consistent_replicate_storage_file.check1 not exists for 2-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) changefeed is working normally rts: 457114316779290664->457114331406925830 checkpoint: 457114316779290664->457114331406925830 run task successfully table test.finish_mark exists wait process cdc.test exit for 1-th time... table batch_add_table.finish_mark not exists for 6-th check, retry later check diff successfully wait process cdc.test exit for 2-th time... go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:32 CST 2025] <<<<<< run test case ddl_only_block_related_table success! >>>>>> table consistent_replicate_storage_file.check1 not exists for 3-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m wait process cdc.test exit for 2-th time... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.20488.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withCredentials [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:33 CST 2025] <<<<<< run test case tidb_mysql_test success! >>>>>> [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } table batch_add_table.finish_mark not exists for 7-th check, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + tso='457114332195454977 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114332195454977 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:46:34 CST 2025] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.2051720519.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file.check1 not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361195130315136 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1033e00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:7134, start at 2025-04-04 15:46:33.34625615 +0800 CST m=+2.270691959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.336 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.336 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361195130315136 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1033e00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:7134, start at 2025-04-04 15:46:33.34625615 +0800 CST m=+2.270691959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.336 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.336 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361200984216734 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1036340007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:7200, start at 2025-04-04 15:46:33.495568632 +0800 CST m=+2.328170794 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.504 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.485 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.485 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table batch_add_table.finish_mark not exists for 8-th check, retry later table consistent_replicate_storage_file.check1 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Fri Apr 4 15:46:37 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.85008502.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:37 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cab0cc-26eb-451f-ba25-75502ac5f662 {"id":"96cab0cc-26eb-451f-ba25-75502ac5f662","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752794} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3d911fd 96cab0cc-26eb-451f-ba25-75502ac5f662 /tidb/cdc/default/default/upstream/7489361165820662906 {"id":7489361165820662906,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cab0cc-26eb-451f-ba25-75502ac5f662 {"id":"96cab0cc-26eb-451f-ba25-75502ac5f662","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752794} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3d911fd 96cab0cc-26eb-451f-ba25-75502ac5f662 /tidb/cdc/default/default/upstream/7489361165820662906 {"id":7489361165820662906,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cab0cc-26eb-451f-ba25-75502ac5f662 {"id":"96cab0cc-26eb-451f-ba25-75502ac5f662","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752794} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3d911fd 96cab0cc-26eb-451f-ba25-75502ac5f662 /tidb/cdc/default/default/upstream/7489361165820662906 {"id":7489361165820662906,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table batch_add_table.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later lease 223195ffc3d911fd revoked table consistent_replicate_storage_file.check1 not exists for 6-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table batch_add_table.finish_mark not exists for 10-th check, retry later table capture_session_done_during_task.t exists check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... empty select id, val from test.availability3 where id=1 run task successfully delete_owner_key pass + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:40 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf {"id":"4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752797} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f2d5 4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf {"id":"4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752797} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f2d5 4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf {"id":"4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752797} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f2d5 4a84a8f8-dfa6-471d-8f8f-c18d7dbebeaf /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 59b51226-3245-4ea2-b1f4-5bdcb1533341 Info: {"upstream_id":7489361195130315136,"namespace":"default","id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:46:40.768591371+08:00","start_ts":457114334183292933,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114334183292933,"checkpoint_ts":457114334183292933,"checkpoint_time":"2025-04-04 15:46:40.687"} wait process cdc.test exit for 1-th time... table consistent_replicate_storage_file.check1 not exists for 7-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6696 0 --:--:-- --:--:-- --:--:-- 6696 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:46:28.332","puller_resolved_ts":"2025-04-04 15:46:22.333","last_synced_ts":"2025-04-04 15:44:12.882","now_ts":"2025-04-04 15:46:29.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:46:28.332","puller_resolved_ts":"2025-04-04' '15:46:22.333","last_synced_ts":"2025-04-04' '15:44:12.882","now_ts":"2025-04-04' '15:46:29.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 8789 9.1 0.0 14080564 147500 ? Sl 15:43 0:14 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 8842 6.1 0.0 14080372 141456 ? Sl 15:43 0:09 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1743752797.1664195,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004300000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752797.1664722,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752797.180568,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046bd180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752797.1806087,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752797.29844,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004646e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752797.2984998,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:46:39.932 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:46:39.932 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489360441792358903] [2025/04/04 15:46:39.932 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:39.932 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:46:39.933 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:39.933 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:46:39.933 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:39.933 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:46:39.933 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:46:39.933 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:39.934 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:46:39.934 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489360441792358903] [2025/04/04 15:46:39.934 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:39.934 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:46:39.935 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:39.935 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:39.935 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:46:39.935 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:46:39.935 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:46:39.935 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:39.935 +08:00] [INFO] [tikv_driver.go:201] ["using API V1."] [2025/04/04 15:46:39.936 +08:00] [INFO] [main.go:180] ["genLock started"] [2025/04/04 15:46:39.937 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=3] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] table batch_add_table.finish_mark not exists for 11-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_resign_owner [Fri Apr 4 15:46:41 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.53455347.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_resign_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_resign_owner.server1 --cluster-id default --addr 127.0.0.1:8300 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later [2025/04/04 15:46:41.936 +08:00] [INFO] [pd.go:441] ["adaptive update ts interval state transition"] [configuredInterval=2s] [prevAdaptiveUpdateInterval=2s] [newAdaptiveUpdateInterval=2s] [requiredStaleness=0s] [prevState=unknown(0)] [newState=normal] table kill_owner_with_ddl.t1 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_file.check1 not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table kill_owner_with_ddl.t1 exists table batch_add_table.finish_mark not exists for 12-th check, retry later check diff successfully check diff failed 1-th time, retry later wait process 8505 exit for 1-th time... {"level":"warn","ts":"2025-04-04T15:46:44.181809+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:44.183697+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:44.292893+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table consistent_replicate_storage_file.check1 not exists for 9-th check, retry later wait process 8505 exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:45 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043aa240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114333935042570 MinTableBarrierTs:457114335259131905 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114333935042570,"min-table-barrier-ts":457114335259131905,"admin-job-type":0} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043aa240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114333935042570 MinTableBarrierTs:457114335259131905 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114333935042570,"min-table-barrier-ts":457114335259131905,"admin-job-type":0} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0043aa240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114333935042570 MinTableBarrierTs:457114335259131905 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114333935042570, checkpointTs: 457114333935042570, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114333935042570,"min-table-barrier-ts":457114335259131905,"admin-job-type":0} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'etcd info' /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8505) - No such process wait process 8505 exit for 3-th time... process 8505 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:46:45 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.86598661.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361235656607268 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10d1a0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:13092, start at 2025-04-04 15:46:43.450306613 +0800 CST m=+2.115949841 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:43.457 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:43.432 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:43.432 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361235656607268 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10d1a0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:13092, start at 2025-04-04 15:46:43.450306613 +0800 CST m=+2.115949841 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:43.457 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:43.432 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:43.432 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361243714393352 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10d1a0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:13167, start at 2025-04-04 15:46:43.472008409 +0800 CST m=+2.034844035 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:43.479 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:43.476 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:43.476 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_reentrant/run.sh using Sink-Type: mysql... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table batch_add_table.finish_mark not exists for 13-th check, retry later check diff successfully "is-owner": true, run task successfully wait process cdc.test exit for 1-th time... table consistent_replicate_storage_file.check1 not exists for 10-th check, retry later wait process cdc.test exit for 2-th time... owner pid: 5350 owner id bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 [Fri Apr 4 15:46:48 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.54625464.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_resign_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_resign_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:47 CST 2025] <<<<<< run test case capture_session_done_during_task success! >>>>>> Starting Upstream TiDB... table batch_add_table.finish_mark not exists for 14-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14412.out cli tso query --pd=http://127.0.0.1:2379 Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:48 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4090 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d41b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336136265732 MinTableBarrierTs:457114336136265732 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336136265732, checkpointTs: 457114336136265732, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1171315-73d0-4ca7-999b-0be51b772162 {"id":"c1171315-73d0-4ca7-999b-0be51b772162","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752805} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f35d c1171315-73d0-4ca7-999b-0be51b772162 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336136265732,"min-table-barrier-ts":457114336136265732,"admin-job-type":0} /tidb/cdc/default/default/task/position/c1171315-73d0-4ca7-999b-0be51b772162/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d41b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336136265732 MinTableBarrierTs:457114336136265732 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336136265732, checkpointTs: 457114336136265732, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1171315-73d0-4ca7-999b-0be51b772162 {"id":"c1171315-73d0-4ca7-999b-0be51b772162","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752805} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f35d c1171315-73d0-4ca7-999b-0be51b772162 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336136265732,"min-table-barrier-ts":457114336136265732,"admin-job-type":0} /tidb/cdc/default/default/task/position/c1171315-73d0-4ca7-999b-0be51b772162/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d41b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336136265732 MinTableBarrierTs:457114336136265732 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336136265732, checkpointTs: 457114336136265732, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1171315-73d0-4ca7-999b-0be51b772162 {"id":"c1171315-73d0-4ca7-999b-0be51b772162","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752805} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f35d c1171315-73d0-4ca7-999b-0be51b772162 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336136265732,"min-table-barrier-ts":457114336136265732,"admin-job-type":0} /tidb/cdc/default/default/task/position/c1171315-73d0-4ca7-999b-0be51b772162/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully start tidb cluster in /tmp/tidb_cdc_test/ddl_reentrant Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table consistent_replicate_storage_file.check1 not exists for 11-th check, retry later + set +x + tso='457114336120537089 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114336120537089 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:46:49 CST 2025] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1445314455.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed table batch_add_table.finish_mark exists table test.t_1 does not exists [2025/04/04 15:46:49.937 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=1] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:49.937 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=2] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:49.937 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=144] [addr=127.0.0.1:9500] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:49.943 +08:00] [INFO] [main.go:196] ["genLock done"] [2025/04/04 15:46:49.943 +08:00] [INFO] [pd_service_discovery.go:551] ["[pd] exit member loop due to context canceled"] [2025/04/04 15:46:49.943 +08:00] [INFO] [resource_manager_client.go:296] ["[resource manager] exit resource token dispatcher"] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_client.go:147] ["[tso] exit tso dispatcher check loop"] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_dispatcher.go:158] ["[tso] exit tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_client.go:157] ["[tso] closing tso client"] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_dispatcher.go:490] ["[tso] exit tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_dispatcher.go:264] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_dispatcher.go:201] ["[tso] exit tso dispatcher"] [dc-location=global] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_stream.go:359] ["tsoStream.recvLoop ended"] [stream=127.0.0.1:2379-1] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241111073742-238d4d79ea31/tso_stream.go:427\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700"] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_client.go:162] ["[tso] close tso client"] [2025/04/04 15:46:49.943 +08:00] [INFO] [tso_client.go:164] ["[tso] tso client is closed"] [2025/04/04 15:46:49.943 +08:00] [INFO] [pd_service_discovery.go:644] ["[pd] close pd service discovery client"] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":"2025-04-04T15:46:50.182379+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:50.184578+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752810.2480652,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004300000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752810.2659557,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046bd180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:50.29412+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... Verifying downstream PD is started... table consistent_replicate_storage_file.check1 not exists for 12-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039b61b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114336569589766 MinTableBarrierTs:457114336831733765 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed {"id":"ac0d7e1d-f5b4-4d89-b233-933ca8c456ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752808} /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e86 ac0d7e1d-f5b4-4d89-b233-933ca8c456ed /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114336569589766,"min-table-barrier-ts":457114336831733765,"admin-job-type":0} /tidb/cdc/default/default/task/position/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039b61b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114336569589766 MinTableBarrierTs:457114336831733765 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed {"id":"ac0d7e1d-f5b4-4d89-b233-933ca8c456ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752808} /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e86 ac0d7e1d-f5b4-4d89-b233-933ca8c456ed /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114336569589766,"min-table-barrier-ts":457114336831733765,"admin-job-type":0} /tidb/cdc/default/default/task/position/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039b61b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114336569589766 MinTableBarrierTs:457114336831733765 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed {"id":"ac0d7e1d-f5b4-4d89-b233-933ca8c456ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752808} /tidb/cdc/default/__cdc_meta__/capture/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 {"id":"bbf91ce1-9fd0-4b98-b815-83fa6d9915a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752802} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e3b bbf91ce1-9fd0-4b98-b815-83fa6d9915a0 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460e86 ac0d7e1d-f5b4-4d89-b233-933ca8c456ed /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114336569589766,"min-table-barrier-ts":457114336831733765,"admin-job-type":0} /tidb/cdc/default/default/task/position/ac0d7e1d-f5b4-4d89-b233-933ca8c456ed/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bbf91ce1-9fd0-4b98-b815-83fa6d9915a0/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "bbf91ce1-9fd0-4b98-b815-83fa6d9915a0" | grep id Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore wait process cdc.test exit for 2-th time... Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:51 CST 2025] <<<<<< run test case batch_add_table success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release "cluster-id": "default" "id": "ac0d7e1d-f5b4-4d89-b233-933ca8c456ed", "cluster-id": "default" run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361270228812978 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1149240004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:7226, start at 2025-04-04 15:46:51.085157733 +0800 CST m=+2.780651643 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:51.093 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:51.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:51.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361270228812978 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1149240004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:7226, start at 2025-04-04 15:46:51.085157733 +0800 CST m=+2.780651643 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:51.093 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:51.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:51.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361267876813187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff114794000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1z1c1-lmh5n-bdb8g, pid:7300, start at 2025-04-04 15:46:50.995828575 +0800 CST m=+2.611471894 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:51.004 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:50.981 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:50.981 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:52 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad1f620b-bfd5-472a-ac22-e8db1a20592f {"id":"ad1f620b-bfd5-472a-ac22-e8db1a20592f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752809} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc419fee9 ad1f620b-bfd5-472a-ac22-e8db1a20592f /tidb/cdc/default/default/upstream/7489361235656607268 {"id":7489361235656607268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad1f620b-bfd5-472a-ac22-e8db1a20592f {"id":"ad1f620b-bfd5-472a-ac22-e8db1a20592f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752809} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc419fee9 ad1f620b-bfd5-472a-ac22-e8db1a20592f /tidb/cdc/default/default/upstream/7489361235656607268 {"id":7489361235656607268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad1f620b-bfd5-472a-ac22-e8db1a20592f {"id":"ad1f620b-bfd5-472a-ac22-e8db1a20592f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752809} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc419fee9 ad1f620b-bfd5-472a-ac22-e8db1a20592f /tidb/cdc/default/default/upstream/7489361235656607268 {"id":7489361235656607268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14505.out cli changefeed create --start-ts=457114336120537089 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table consistent_replicate_storage_file.check1 not exists for 13-th check, retry later Create changefeed successfully! ID: 38858f9a-208d-4d72-a03b-6c7d89f11520 Info: {"upstream_id":7489361235656607268,"namespace":"default","id":"38858f9a-208d-4d72-a03b-6c7d89f11520","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:46:52.974603088+08:00","start_ts":457114336120537089,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114336120537089,"checkpoint_ts":457114336120537089,"checkpoint_time":"2025-04-04 15:46:48.077"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 capture_id: ac0d7e1d-f5b4-4d89-b233-933ca8c456ed + '[' mysql == mysql ']' + check_table_exists test.t2 127.0.0.1 3306 table test.t2 not exists for 1-th check, retry later table test.t2 exists + sleep 10 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 2 100 2 0 0 666 0 --:--:-- --:--:-- --:--:-- 666 {}cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep ac0d7e1d-f5b4-4d89-b233-933ca8c456ed -A1 | grep '"is-owner": true' 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 + set +x table savepoint.finish_mark not exists for 1-th check, retry later [Fri Apr 4 15:46:54 CST 2025] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.86568658.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 wait process 8664 exit for 1-th time... table consistent_replicate_storage_file.check1 not exists for 14-th check, retry later "is-owner": true, run task successfully test_resign_owner: pass pass test: get status pass test: health [2025/04/04 15:46:51.246 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info=null] [2025/04/04 15:46:53.796 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info="{\"upstream_id\":7489360319884375937,\"namespace\":\"test\",\"id\":\"changefeed-test-v2-black-hole-1\",\"sink_uri\":\"blackhole:\",\"create_time\":\"2025-04-04T15:46:51.212598076+08:00\",\"start_ts\":457114336933445635,\"config\":{\"memory_quota\":1073741824,\"case_sensitive\":false,\"force_replicate\":false,\"ignore_ineligible_table\":false,\"check_gc_safe_point\":true,\"enable_sync_point\":false,\"bdr_mode\":false,\"sync_point_interval\":600000000000,\"sync_point_retention\":86400000000000,\"filter\":{\"rules\":[\"*.*\"]},\"mounter\":{\"worker_num\":16},\"sink\":{\"csv\":{\"delimiter\":\",\",\"quote\":\"\\\"\",\"null\":\"\\\\N\",\"include_commit_ts\":false},\"transaction_atomicity\":\"\",\"encoder_concurrency\":32,\"terminator\":\"\\r\\n\",\"date_separator\":\"day\",\"enable_partition_separator\":true,\"content_compatible\":false,\"send_bootstrap_interval_in_sec\":120,\"send_bootstrap_in_msg_count\":10000,\"send_bootstrap_to_all_partition\":true,\"debezium_disable_schema\":false,\"debezium\":{\"output_old_value\":true},\"open\":{\"output_old_value\":true}},\"consistent\":{\"level\":\"none\",\"max_log_size\":64,\"flush_interval\":2000,\"meta_flush_interval\":200,\"encoding_worker_num\":16,\"flush_worker_num\":8,\"storage\":\"\",\"use_file_backend\":false},\"scheduler\":{\"enable_table_across_nodes\":false,\"region_threshold\":100000,\"write_key_threshold\":0},\"integrity\":{\"integrity_check_level\":\"none\",\"corruption_handle_level\":\"warn\"}},\"state\":\"normal\",\"creator_version\":\"v9.0.0-beta.1-10-ge8bfbe4ba\",\"resolved_ts\":0,\"checkpoint_ts\":457114337418412036,\"checkpoint_time\":\"2025-04-04 15:46:53.028\"}"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361285583163708 TiDB Cluster ID. tikv_gc_leader_uuid 657ff117448000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:9637, start at 2025-04-04 15:46:53.861385036 +0800 CST m=+2.498191323 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:53.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:53.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:53.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361285583163708 TiDB Cluster ID. tikv_gc_leader_uuid 657ff117448000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:9637, start at 2025-04-04 15:46:53.861385036 +0800 CST m=+2.498191323 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:53.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:53.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:53.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361282906720500 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1175940002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9m9ct-6m5th-1p46x, pid:9711, start at 2025-04-04 15:46:53.929941131 +0800 CST m=+2.483547060 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:53.938 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:53.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:53.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process 8664 exit for 2-th time... wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:46:56.183236+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:56.186529+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:46:56.294995+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8664) - No such process wait process 8664 exit for 3-th time... process 8664 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:46:56 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.88388840.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table savepoint.finish_mark not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_kill_capture [Fri Apr 4 15:46:57 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.56365638.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file.check1 not exists for 15-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:57 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b {"id":"1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752815} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4355ad7 1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b /tidb/cdc/default/default/upstream/7489361270228812978 {"id":7489361270228812978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b {"id":"1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752815} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4355ad7 1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b /tidb/cdc/default/default/upstream/7489361270228812978 {"id":7489361270228812978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b {"id":"1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752815} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4355ad7 1f91a9bb-7dbf-474f-8fae-8ac49ae3ad8b /tidb/cdc/default/default/upstream/7489361270228812978 {"id":7489361270228812978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table test.t2 not exists for 1-th check, retry later check_changefeed_status 127.0.0.1:8300 496b94f8-8629-4cdf-ba90-c7f6883c009c normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=496b94f8-8629-4cdf-ba90-c7f6883c009c + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/496b94f8-8629-4cdf-ba90-c7f6883c009c/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1323 0 --:--:-- --:--:-- --:--:-- 1303 100 86 100 86 0 0 1323 0 --:--:-- --:--:-- --:--:-- 1303 + info='{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' + echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' {"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065} ++ echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 496b94f8-8629-4cdf-ba90-c7f6883c009c normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=496b94f8-8629-4cdf-ba90-c7f6883c009c + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/496b94f8-8629-4cdf-ba90-c7f6883c009c/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ***************** properties ***************** "readproportion"="0" "recordcount"="20" "dotransactions"="false" "threadcount"="4" "scanproportion"="0" "mysql.db"="changefeed_auto_stop_1" "workload"="core" "readallfields"="true" "operationcount"="0" "requestdistribution"="uniform" "mysql.user"="root" "updateproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" ********************************************** Run finished, takes 10.542033ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3499.3, Avg(us): 1983, Min(us): 987, Max(us): 5005, 95th(us): 5000, 99th(us): 6000 ***************** properties ***************** "scanproportion"="0" "updateproportion"="0" "mysql.user"="root" "mysql.host"="127.0.0.1" "mysql.port"="4000" "insertproportion"="0" "mysql.db"="changefeed_auto_stop_2" "threadcount"="4" "readallfields"="true" "recordcount"="20" "workload"="core" "operationcount"="0" "requestdistribution"="uniform" "dotransactions"="false" "readproportion"="0" ********************************************** Run finished, takes 11.235188ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2907.1, Avg(us): 2067, Min(us): 1151, Max(us): 4552, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "dotransactions"="false" "mysql.port"="4000" "mysql.user"="root" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_auto_stop_3" "operationcount"="0" "insertproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "readproportion"="0" "threadcount"="4" "recordcount"="20" "readallfields"="true" "workload"="core" "updateproportion"="0" ********************************************** Run finished, takes 10.570841ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3016.7, Avg(us): 1912, Min(us): 1090, Max(us): 3856, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "mysql.port"="4000" "updateproportion"="0" "operationcount"="0" "mysql.host"="127.0.0.1" "threadcount"="4" "recordcount"="20" "readproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "scanproportion"="0" "dotransactions"="false" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_4" "readallfields"="true" "workload"="core" ********************************************** Run finished, takes 10.400989ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2923.1, Avg(us): 1777, Min(us): 1042, Max(us): 3474, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:46:58 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1119211194.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1131 0 --:--:-- --:--:-- --:--:-- 1131 + info='{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' + echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' {"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065} ++ echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":457114338703704065,"checkpoint_ts":457114338703704065}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361290838750676 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119b40000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:6536, start at 2025-04-04 15:46:56.34134747 +0800 CST m=+2.284215047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.349 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.336 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.336 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361290838750676 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119b40000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:6536, start at 2025-04-04 15:46:56.34134747 +0800 CST m=+2.284215047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.349 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.336 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.336 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361295079844701 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119b40000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:6594, start at 2025-04-04 15:46:56.375423159 +0800 CST m=+2.236644589 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.383 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.385 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.385 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:46:58.547 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info="{\"upstream_id\":7489360319884375937,\"namespace\":\"test\",\"id\":\"changefeed-test-v2-black-hole-1\",\"sink_uri\":\"blackhole:\",\"create_time\":\"2025-04-04T15:46:51.212598076+08:00\",\"start_ts\":457114336933445635,\"admin_job_type\":1,\"config\":{\"memory_quota\":1123450,\"case_sensitive\":false,\"force_replicate\":false,\"ignore_ineligible_table\":false,\"check_gc_safe_point\":false,\"enable_sync_point\":false,\"bdr_mode\":false,\"sync_point_interval\":600000000000,\"sync_point_retention\":86400000000000,\"filter\":{\"rules\":[\"a.d\",\"b.x\"],\"ignore_txn_start_ts\":[1,2,3],\"event_filters\":[{\"matcher\":[\"test.worker\"],\"ignore_event\":[\"update\"],\"ignore_sql\":[\"^drop\",\"add column\"],\"ignore_insert_value_expr\":\"id \\u003e= 100\",\"ignore_update_new_value_expr\":\"gender = 'male'\",\"ignore_update_old_value_expr\":\"age \\u003c 18\",\"ignore_delete_value_expr\":\"id \\u003e 100\"}]},\"mounter\":{\"worker_num\":17},\"sink\":{\"protocol\":\"arvo\",\"csv\":{\"delimiter\":\",\",\"quote\":\"\\\"\",\"null\":\"\\\\N\",\"include_commit_ts\":false},\"column_selectors\":[{\"matcher\":[\"a.b\"],\"columns\":[\"c\"]}],\"transaction_atomicity\":\"table\",\"encoder_concurrency\":32,\"terminator\":\"a\",\"date_separator\":\"day\",\"enable_partition_separator\":true,\"content_compatible\":true,\"send_bootstrap_interval_in_sec\":120,\"send_bootstrap_in_msg_count\":10000,\"send_bootstrap_to_all_partition\":true,\"debezium_disable_schema\":true,\"debezium\":{\"output_old_value\":true},\"open\":{\"output_old_value\":true}},\"consistent\":{\"level\":\"none\",\"max_log_size\":64,\"flush_interval\":2000,\"meta_flush_interval\":200,\"encoding_worker_num\":31,\"flush_worker_num\":18,\"storage\":\"\",\"use_file_backend\":false},\"scheduler\":{\"enable_table_across_nodes\":false,\"region_threshold\":13,\"write_key_threshold\":0},\"integrity\":{\"integrity_check_level\":\"none\",\"corruption_handle_level\":\"warn\"}},\"state\":\"stopped\",\"creator_version\":\"v9.0.0-beta.1-10-ge8bfbe4ba\",\"resolved_ts\":457114337418412036,\"checkpoint_ts\":457114337418412036,\"checkpoint_time\":\"2025-04-04 15:46:53.028\"}"] table savepoint.finish_mark exists check diff successfully table test.t2 not exists for 2-th check, retry later table consistent_replicate_storage_file.check1 not exists for 16-th check, retry later wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00153cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5bb78f4d-a6bc-43ea-a5e7-13f20de39178 {"id":"5bb78f4d-a6bc-43ea-a5e7-13f20de39178","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752816} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f3ee 5bb78f4d-a6bc-43ea-a5e7-13f20de39178 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/5bb78f4d-a6bc-43ea-a5e7-13f20de39178/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00153cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5bb78f4d-a6bc-43ea-a5e7-13f20de39178 {"id":"5bb78f4d-a6bc-43ea-a5e7-13f20de39178","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752816} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f3ee 5bb78f4d-a6bc-43ea-a5e7-13f20de39178 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/5bb78f4d-a6bc-43ea-a5e7-13f20de39178/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00153cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5bb78f4d-a6bc-43ea-a5e7-13f20de39178 {"id":"5bb78f4d-a6bc-43ea-a5e7-13f20de39178","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752816} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f3ee 5bb78f4d-a6bc-43ea-a5e7-13f20de39178 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/5bb78f4d-a6bc-43ea-a5e7-13f20de39178/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:00 CST 2025] <<<<<< run test case savepoint success! >>>>>> table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 skip some DDLs in tidb v4.0.x [Fri Apr 4 15:47:00 CST 2025] <<<<<< START cdc server in ddl_reentrant case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_reentrant.79697971.out server --log-file /tmp/tidb_cdc_test/ddl_reentrant/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_reentrant/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040ac7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114338928885766 MinTableBarrierTs:457114339191029766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114339191029764, checkpointTs: 457114339191029764, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114338928885766,"min-table-barrier-ts":457114339191029766,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040ac7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114338928885766 MinTableBarrierTs:457114339191029766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114339191029764, checkpointTs: 457114339191029764, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114338928885766,"min-table-barrier-ts":457114339191029766,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040ac7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114338928885766 MinTableBarrierTs:457114339191029766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114339191029764, checkpointTs: 457114339191029764, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114339191029766, checkpointTs: 457114339191029766, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114338928885766,"min-table-barrier-ts":457114339191029766,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + run_sql 'SELECT * from test.t2' 127.0.0.1 3306 + check_contains 'a: 2023-02-08' + stop_tidb_cluster table test.t2 not exists for 3-th check, retry later table consistent_replicate_storage_file.check1 not exists for 17-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 pass test: changefeed apis [2025/04/04 15:47:01.146 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info=null] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:01 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:47:01 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1125111253.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "is-owner": true, run task successfully ____________________________________ *************************** 1. row *************************** primary_ts: 457114276331520000 secondary_ts: 457114279130169351 *************************** 2. row *************************** primary_ts: 457114284195840000 secondary_ts: 457114284425478145 *************************** 3. row *************************** primary_ts: 457114292060160000 secondary_ts: 457114292289536001 *************************** 4. row *************************** primary_ts: 457114299924480000 secondary_ts: 457114300154118145 *************************** 5. row *************************** primary_ts: 457114307788800000 secondary_ts: 457114308044390401 *************************** 6. row *************************** primary_ts: 457114315653120000 secondary_ts: 457114315908710401 *************************** 7. row *************************** primary_ts: 457114323517440000 secondary_ts: 457114323799506949 *************************** 8. row *************************** primary_ts: 457114331381760000 secondary_ts: 457114331663826949 skip invalid syncpoint primary_ts: 457114276331520000, first_ddl_ts: 457114312632434707 skip invalid syncpoint primary_ts: 457114284195840000, first_ddl_ts: 457114312632434707 skip invalid syncpoint primary_ts: 457114292060160000, first_ddl_ts: 457114312632434707 skip invalid syncpoint primary_ts: 457114299924480000, first_ddl_ts: 457114312632434707 skip invalid syncpoint primary_ts: 457114307788800000, first_ddl_ts: 457114312632434707 check diff successfully check diff successfully check diff successfully check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:57 CST 2025] <<<<<< run test case syncpoint success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:02.184362+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:02.187525+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:02.296816+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table test.t2 not exists for 4-th check, retry later table sink_hang.t2 exists check diff failed 1-th time, retry later owner pid: 5641 owner id 1451c598-5378-4af7-aa61-47c17fad7103 table test.availability1 exists nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table consistent_replicate_storage_file.check1 not exists for 18-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 {"id":"8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752820} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc449afdb 8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 /tidb/cdc/default/default/upstream/7489361290838750676 {"id":7489361290838750676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 {"id":"8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752820} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc449afdb 8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 /tidb/cdc/default/default/upstream/7489361290838750676 {"id":7489361290838750676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 {"id":"8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752820} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc449afdb 8d4fa39a-ffb9-43fa-8c86-b28cffd2dc74 /tidb/cdc/default/default/upstream/7489361290838750676 {"id":7489361290838750676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ------------------------------------------ test ddl create database ddl_reentrant, is_reentrant: false restored_sql: CREATE DATABASE `ddl_reentrant` pass test: delete changefeed apis pass test: capture apis pass test: processor apis 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:04.170103+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:04.170146+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:04.170734+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:04.17078+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:04.282213+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:04.282247+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:04 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd1dfc8-9573-4946-9ec8-24ea8aabb62b {"id":"7cd1dfc8-9573-4946-9ec8-24ea8aabb62b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752821} /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6851 7cd1dfc8-9573-4946-9ec8-24ea8aabb62b /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd1dfc8-9573-4946-9ec8-24ea8aabb62b {"id":"7cd1dfc8-9573-4946-9ec8-24ea8aabb62b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752821} /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6851 7cd1dfc8-9573-4946-9ec8-24ea8aabb62b /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd1dfc8-9573-4946-9ec8-24ea8aabb62b {"id":"7cd1dfc8-9573-4946-9ec8-24ea8aabb62b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752821} /tidb/cdc/default/__cdc_meta__/capture/a6b0f84b-c834-4091-b32d-09c3e7891e36 {"id":"a6b0f84b-c834-4091-b32d-09c3e7891e36","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752818} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6816 a6b0f84b-c834-4091-b32d-09c3e7891e36 /tidb/cdc/default/__cdc_meta__/owner/223195ffc43d6851 7cd1dfc8-9573-4946-9ec8-24ea8aabb62b /tidb/cdc/default/default/upstream/7489361285583163708 {"id":7489361285583163708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 5714681d-0be4-4b2c-b60a-a8e259a57685 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=5714681d-0be4-4b2c-b60a-a8e259a57685 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 5714681d-0be4-4b2c-b60a-a8e259a57685 -s + info='{ "upstream_id": 7489361285583163708, "namespace": "default", "id": "5714681d-0be4-4b2c-b60a-a8e259a57685", "state": "normal", "checkpoint_tso": 457114338680373257, "checkpoint_time": "2025-04-04 15:46:57.842", "error": null }' + echo '{ "upstream_id": 7489361285583163708, "namespace": "default", "id": "5714681d-0be4-4b2c-b60a-a8e259a57685", "state": "normal", "checkpoint_tso": 457114338680373257, "checkpoint_time": "2025-04-04 15:46:57.842", "error": null }' { "upstream_id": 7489361285583163708, "namespace": "default", "id": "5714681d-0be4-4b2c-b60a-a8e259a57685", "state": "normal", "checkpoint_tso": 457114338680373257, "checkpoint_time": "2025-04-04 15:46:57.842", "error": null } ++ echo '{' '"upstream_id":' 7489361285583163708, '"namespace":' '"default",' '"id":' '"5714681d-0be4-4b2c-b60a-a8e259a57685",' '"state":' '"normal",' '"checkpoint_tso":' 457114338680373257, '"checkpoint_time":' '"2025-04-04' '15:46:57.842",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361285583163708, '"namespace":' '"default",' '"id":' '"5714681d-0be4-4b2c-b60a-a8e259a57685",' '"state":' '"normal",' '"checkpoint_tso":' 457114338680373257, '"checkpoint_time":' '"2025-04-04' '15:46:57.842",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... [Fri Apr 4 15:47:05 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.57815783.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t2 not exists for 5-th check, retry later Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table consistent_replicate_storage_file.check1 exists table consistent_replicate_storage_file.t2 not exists for 1-th check, retry later changefeed is working normally rts: 457114340199235589->457114340579344389 checkpoint: 457114340199235589->457114340579344389 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:47:05.214 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [duration=26.583011ms] [sql="CREATE DATABASE `ddl_reentrant`"] run task successfully Starting Upstream TiDB... resume changefeed 08831740-2868-468d-87ae-f758f41a6164 from 457114340225450004 check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint_check_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table test.t2 not exists for 6-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... pass test: owner apis pass test: set log level table consistent_replicate_storage_file.t2 not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... + cleanup_process cdc.test wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:08 CST 2025] <<<<<< run test case sink_hang success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:08.186+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:08.188624+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:08.29718+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:08 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00439e750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114341288181769 MinTableBarrierTs:457114341288181770 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/capture/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 {"id":"2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f75 2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114341288181769,"min-table-barrier-ts":457114341288181770,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00439e750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114341288181769 MinTableBarrierTs:457114341288181770 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/capture/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 {"id":"2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f75 2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114341288181769,"min-table-barrier-ts":457114341288181770,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00439e750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114341288181769 MinTableBarrierTs:457114341288181770 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1451c598-5378-4af7-aa61-47c17fad7103 {"id":"1451c598-5378-4af7-aa61-47c17fad7103","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752817} /tidb/cdc/default/__cdc_meta__/capture/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 {"id":"2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f0c 1451c598-5378-4af7-aa61-47c17fad7103 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1460f75 2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114341288181769,"min-table-barrier-ts":457114341288181770,"admin-job-type":0} /tidb/cdc/default/default/task/position/1451c598-5378-4af7-aa61-47c17fad7103/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "1451c598-5378-4af7-aa61-47c17fad7103" | grep id changefeed is working normally rts: 457114340225450004->457114341103632390 checkpoint: 457114340225450004->457114341365776388 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:47:07.194 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114340225450001] [ddl="CREATE DATABASE `ddl_reentrant`"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [error="Error 1007 (HY000): Can't create database 'ddl_reentrant'; database exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t3 (a int primary key, b int) partition by range(a) (partition p0 values less than (1000), partition p1 values less than (2000)), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000)) wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 table test.t2 not exists for 7-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 4 15:47:09 CST 2025] <<<<<< run test case api_v2 success! >>>>>>' [Fri Apr 4 15:47:09 CST 2025] <<<<<< run test case api_v2 success! >>>>>> + stop_tidb_cluster table consistent_replicate_storage_file.t2 not exists for 3-th check, retry later table changefeed_auto_stop_2.usertable not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used wait process 8843 exit for 1-th time... wait process 8843 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8843) - No such process wait process 8843 exit for 3-th time... process 8843 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:47:09 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.89828984.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "cluster-id": "default" "id": "2d9ac427-49a2-4fd9-a25f-43f9e8a2b0b1", "cluster-id": "default" run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/syncpoint_check_ts Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore changefeed is working normally rts: 457114341365776388->457114341444419587 checkpoint: 457114341444419587->457114341444419587 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:47:09.163 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [duration=25.012218ms] [sql="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] run task successfully table test.t2 not exists for 8-th check, retry later table consistent_replicate_storage_file.t2 not exists for 4-th check, retry later Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) resume changefeed 08831740-2868-468d-87ae-f758f41a6164 from 457114341444419587 check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361350344416550 TiDB Cluster ID. tikv_gc_leader_uuid 657ff125f14000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:21806, start at 2025-04-04 15:47:08.914881692 +0800 CST m=+2.113928368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:08.959 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:08.919 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:08.919 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361350344416550 TiDB Cluster ID. tikv_gc_leader_uuid 657ff125f14000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:21806, start at 2025-04-04 15:47:08.914881692 +0800 CST m=+2.113928368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:08.959 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:08.919 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:08.919 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361339814561432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1263b00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-9ffrv-vdlp8-j5hfw, pid:21870, start at 2025-04-04 15:47:09.167399243 +0800 CST m=+2.286394363 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:09.175 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:09.164 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:09.164 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:47:05.029 +08:00] [INFO] [case.go:115] ["sync updatePKUK take: 11.447059341s"] Debugger for raftstore-v2 is used 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1743752832.1672401,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004300000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752832.1672943,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752832.1815505,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046bd180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752832.1815915,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":1743752832.299423,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004646e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752832.2994654,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0018f4fc0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74544781-0313-42ce-96a0-3f73500c1977 {"id":"74544781-0313-42ce-96a0-3f73500c1977","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752829} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f47e 74544781-0313-42ce-96a0-3f73500c1977 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/74544781-0313-42ce-96a0-3f73500c1977/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0018f4fc0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74544781-0313-42ce-96a0-3f73500c1977 {"id":"74544781-0313-42ce-96a0-3f73500c1977","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752829} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f47e 74544781-0313-42ce-96a0-3f73500c1977 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/74544781-0313-42ce-96a0-3f73500c1977/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0018f4fc0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74544781-0313-42ce-96a0-3f73500c1977 {"id":"74544781-0313-42ce-96a0-3f73500c1977","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752829} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f47e 74544781-0313-42ce-96a0-3f73500c1977 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/74544781-0313-42ce-96a0-3f73500c1977/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task failed 1-th time, retry later table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.23120.out cli tso query --pd=http://127.0.0.1:2379 start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... table test.t2 not exists for 9-th check, retry later Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table consistent_replicate_storage_file.t2 not exists for 5-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:14.187216+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:14.189427+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:14.297643+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table changefeed_auto_stop_4.usertable exists + set +x + tso='457114342777159682 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114342777159682 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:47:14 CST 2025] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.2314823150.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) check diff successfully [Pipeline] // cache table test.t2 not exists for 10-th check, retry later [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // dir table consistent_replicate_storage_file.t2 exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 [Pipeline] // withCredentials [Pipeline] } check diff successfully wait process cdc.test exit for 1-th time... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // podTemplate Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // withEnv [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:16 CST 2025] <<<<<< run test case changefeed_auto_stop success! >>>>>> [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... changefeed is working normally rts: 457114341444419587->457114343462928390 checkpoint: 457114341444419587->457114343462928390 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:47:13.238 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114341431312413] [ddl="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [error="Error 1050 (42S01): Table 'ddl_reentrant.t3' already exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t4 (a int primary key, b int), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT) table test.t2 not exists for 11-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:47:17 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1148211484.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:17 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1382a664-74ce-45e9-b8af-9a65fcfa573a {"id":"1382a664-74ce-45e9-b8af-9a65fcfa573a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752835} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc477b9e6 1382a664-74ce-45e9-b8af-9a65fcfa573a /tidb/cdc/default/default/upstream/7489361350344416550 {"id":7489361350344416550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1382a664-74ce-45e9-b8af-9a65fcfa573a {"id":"1382a664-74ce-45e9-b8af-9a65fcfa573a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752835} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc477b9e6 1382a664-74ce-45e9-b8af-9a65fcfa573a /tidb/cdc/default/default/upstream/7489361350344416550 {"id":7489361350344416550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1382a664-74ce-45e9-b8af-9a65fcfa573a {"id":"1382a664-74ce-45e9-b8af-9a65fcfa573a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752835} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc477b9e6 1382a664-74ce-45e9-b8af-9a65fcfa573a /tidb/cdc/default/default/upstream/7489361350344416550 {"id":7489361350344416550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.23199.out cli changefeed create --start-ts=457114342777159682 --sink-uri=mysql://root@127.0.0.1:3306/ Create changefeed successfully! ID: 88c7fb36-a4bd-4746-ba18-62a3cc355e99 Info: {"upstream_id":7489361350344416550,"namespace":"default","id":"88c7fb36-a4bd-4746-ba18-62a3cc355e99","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:47:18.379000454+08:00","start_ts":457114342777159682,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114342777159682,"checkpoint_ts":457114342777159682,"checkpoint_time":"2025-04-04 15:47:13.470"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 table test.t2 not exists for 12-th check, retry later [Pipeline] // withEnv ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } changefeed is working normally rts: 457114343725334533->457114343725334533 checkpoint: 457114343711965189->457114343725334533 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:47:17.362 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [duration=25.216615ms] [sql="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] run task successfully [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) resume changefeed 08831740-2868-468d-87ae-f758f41a6164 from 457114343711965189 check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 35 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361387064236600 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12f5f4002a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:12428, start at 2025-04-04 15:47:18.532327248 +0800 CST m=+2.399021953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:18.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:18.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:18.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361387064236600 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12f5f4002a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:12428, start at 2025-04-04 15:47:18.532327248 +0800 CST m=+2.399021953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:18.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:18.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:18.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... {"level":"warn","ts":"2025-04-04T15:47:20.18899+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020048c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:20.190294+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e0c700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:20.298941+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e05c00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361393645904695 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12f638000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:12499, start at 2025-04-04 15:47:18.568807288 +0800 CST m=+2.337734072 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:18.576 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:18.542 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:18.542 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... wait process 8987 exit for 1-th time... table test.t2 exists wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:21 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {UpstreamID:7489361033720874216 Namespace:default ID:ec8b9d44-a91a-4b53-b4fc-f954b18f6437 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:46:24.416919945 +0800 CST StartTs:457114329895927812 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004305560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114329908772870} {CheckpointTs:457114344444657667 MinTableBarrierTs:457114344444657667 AdminJobType:noop} span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a367362c-02a7-4df5-988c-978ba48198b2 {"id":"a367362c-02a7-4df5-988c-978ba48198b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752838} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8a49 a367362c-02a7-4df5-988c-978ba48198b2 /tidb/cdc/default/default/changefeed/info/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"upstream-id":7489361033720874216,"namespace":"default","changefeed-id":"ec8b9d44-a91a-4b53-b4fc-f954b18f6437","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:46:24.416919945+08:00","start-ts":457114329895927812,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114329908772870} /tidb/cdc/default/default/changefeed/status/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":457114344444657667,"min-table-barrier-ts":457114344444657667,"admin-job-type":0} /tidb/cdc/default/default/task/position/a367362c-02a7-4df5-988c-978ba48198b2/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {UpstreamID:7489361033720874216 Namespace:default ID:ec8b9d44-a91a-4b53-b4fc-f954b18f6437 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:46:24.416919945 +0800 CST StartTs:457114329895927812 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004305560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114329908772870} {CheckpointTs:457114344444657667 MinTableBarrierTs:457114344444657667 AdminJobType:noop} span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a367362c-02a7-4df5-988c-978ba48198b2 {"id":"a367362c-02a7-4df5-988c-978ba48198b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752838} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8a49 a367362c-02a7-4df5-988c-978ba48198b2 /tidb/cdc/default/default/changefeed/info/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"upstream-id":7489361033720874216,"namespace":"default","changefeed-id":"ec8b9d44-a91a-4b53-b4fc-f954b18f6437","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:46:24.416919945+08:00","start-ts":457114329895927812,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114329908772870} /tidb/cdc/default/default/changefeed/status/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":457114344444657667,"min-table-barrier-ts":457114344444657667,"admin-job-type":0} /tidb/cdc/default/default/task/position/a367362c-02a7-4df5-988c-978ba48198b2/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {UpstreamID:7489361033720874216 Namespace:default ID:ec8b9d44-a91a-4b53-b4fc-f954b18f6437 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:46:24.416919945 +0800 CST StartTs:457114329895927812 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004305560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114329908772870} {CheckpointTs:457114344444657667 MinTableBarrierTs:457114344444657667 AdminJobType:noop} span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114344444657667, checkpointTs: 457114344444657667, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114344444657668, checkpointTs: 457114344444657667, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a367362c-02a7-4df5-988c-978ba48198b2 {"id":"a367362c-02a7-4df5-988c-978ba48198b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752838} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc35d8a49 a367362c-02a7-4df5-988c-978ba48198b2 /tidb/cdc/default/default/changefeed/info/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"upstream-id":7489361033720874216,"namespace":"default","changefeed-id":"ec8b9d44-a91a-4b53-b4fc-f954b18f6437","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:46:24.416919945+08:00","start-ts":457114329895927812,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114329908772870} /tidb/cdc/default/default/changefeed/status/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":457114344444657667,"min-table-barrier-ts":457114344444657667,"admin-job-type":0} /tidb/cdc/default/default/task/position/a367362c-02a7-4df5-988c-978ba48198b2/ec8b9d44-a91a-4b53-b4fc-f954b18f6437 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361033720874216 {"id":7489361033720874216,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... wait process 8987 exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361394355093234 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1308f8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:15484, start at 2025-04-04 15:47:19.768928808 +0800 CST m=+2.100627872 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:19.776 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:19.742 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:19.742 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361394355093234 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1308f8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:15484, start at 2025-04-04 15:47:19.768928808 +0800 CST m=+2.100627872 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:19.776 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:19.742 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:19.742 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361397096453628 TiDB Cluster ID. tikv_gc_leader_uuid 657ff130b4c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:15548, start at 2025-04-04 15:47:19.907059428 +0800 CST m=+2.155500272 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:19.936 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:19.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:19.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) run task failed 1-th time, retry later wait process cdc.test exit for 1-th time... Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8987) - No such process wait process 8987 exit for 3-th time... process 8987 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:47:22 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.91339135.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:22 CST 2025] <<<<<< run test case resolve_lock success! >>>>>> + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.cli.13725.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 + set +x + tso='457114345247604737 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114345247604737 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:47:24 CST 2025] <<<<<< START cdc server in syncpoint_check_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.1375813760.out server --log-file /tmp/tidb_cdc_test/syncpoint_check_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint_check_ts/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 nonempty select id, val from test.availability1 where id=2 and val=2 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16933.out cli tso query --pd=http://127.0.0.1:2379 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_capture [Fri Apr 4 15:47:24 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.59825984.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/http_api_tls_with_user_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... changefeed is working normally rts: 457114345297936390->457114345560080392 checkpoint: 457114345297936390->457114345560080392 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:47:20.246 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114343711965185] [ddl="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [error="Error 1050 (42S01): Table 'ddl_reentrant.t4' already exists"] run task successfully ------------------------------------------ test ddl alter table ddl_reentrant.t3 exchange partition p0 with table ddl_reentrant.t4 without validation, is_reentrant: true restored_sql: ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4088 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003508630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 {"id":"fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752842} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f507 fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003508630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 {"id":"fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752842} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f507 fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003508630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336280182807 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 {"id":"fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752842} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f507 fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336280182807,"admin-job-type":0} /tidb/cdc/default/default/task/position/fbe8c3b5-8a70-4b69-ac1d-3d9be7ace8a9/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully + set +x + tso='457114345640296449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114345640296449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114345640296449 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:47:25 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1696016962.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [2025/04/04 15:47:26.165 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:35' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.166 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:46' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.166 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:47' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.168 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:48' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.226 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:58:29' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.341 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:50' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.341 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:48' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.343 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:47' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.344 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:49' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.348 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:01' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.356 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:57:42' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.426 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:58:11' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.432 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:58:22' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.445 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:58:03' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.549 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:47:26.565 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2025/04/04 15:47:27.052 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // node [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:27 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a397849-da50-47bb-a2b3-9212d8a8af76 {"id":"4a397849-da50-47bb-a2b3-9212d8a8af76","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752844} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49e15e8 4a397849-da50-47bb-a2b3-9212d8a8af76 /tidb/cdc/default/default/upstream/7489361387064236600 {"id":7489361387064236600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a397849-da50-47bb-a2b3-9212d8a8af76 {"id":"4a397849-da50-47bb-a2b3-9212d8a8af76","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752844} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49e15e8 4a397849-da50-47bb-a2b3-9212d8a8af76 /tidb/cdc/default/default/upstream/7489361387064236600 {"id":7489361387064236600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a397849-da50-47bb-a2b3-9212d8a8af76 {"id":"4a397849-da50-47bb-a2b3-9212d8a8af76","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752844} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49e15e8 4a397849-da50-47bb-a2b3-9212d8a8af76 /tidb/cdc/default/default/upstream/7489361387064236600 {"id":7489361387064236600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.cli.13824.out cli changefeed create --start-ts=457114345247604737 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint_check_ts/conf/changefeed.toml [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } Create changefeed successfully! ID: 4917c156-231e-493e-b31e-114ee2d2c964 Info: {"upstream_id":7489361387064236600,"namespace":"default","id":"4917c156-231e-493e-b31e-114ee2d2c964","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:47:27.850187519+08:00","start_ts":457114345247604737,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114345247604737,"checkpoint_ts":457114345247604737,"checkpoint_time":"2025-04-04 15:47:22.894"} PASS [Pipeline] // stage [Pipeline] } changefeed is working normally rts: 457114345822224393->457114346346512391 checkpoint: 457114345822224393->457114346346512391 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:47:26.430 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [duration=40.096312ms] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION"] run task successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004794b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114346268917768 MinTableBarrierTs:457114346544168963 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114346544168961, checkpointTs: 457114346282024962, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114346268917768,"min-table-barrier-ts":457114346544168963,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004794b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114346268917768 MinTableBarrierTs:457114346544168963 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114346544168961, checkpointTs: 457114346282024962, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114346268917768,"min-table-barrier-ts":457114346544168963,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004794b40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114346268917768 MinTableBarrierTs:457114346544168963 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114346544168963, checkpointTs: 457114346282024962, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114346544168961, checkpointTs: 457114346282024962, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114346268917768,"min-table-barrier-ts":457114346544168963,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' resume changefeed 08831740-2868-468d-87ae-f758f41a6164 from 457114345927082001 check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:28 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6918cb14-51a4-4957-abd2-e78719845af7 {"id":"6918cb14-51a4-4957-abd2-e78719845af7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4aa00e4 6918cb14-51a4-4957-abd2-e78719845af7 /tidb/cdc/default/default/upstream/7489361394355093234 {"id":7489361394355093234,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6918cb14-51a4-4957-abd2-e78719845af7 {"id":"6918cb14-51a4-4957-abd2-e78719845af7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4aa00e4 6918cb14-51a4-4957-abd2-e78719845af7 /tidb/cdc/default/default/upstream/7489361394355093234 {"id":7489361394355093234,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6918cb14-51a4-4957-abd2-e78719845af7 {"id":"6918cb14-51a4-4957-abd2-e78719845af7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4aa00e4 6918cb14-51a4-4957-abd2-e78719845af7 /tidb/cdc/default/default/upstream/7489361394355093234 {"id":7489361394355093234,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114345640296449 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.17022.out cli changefeed create --start-ts=457114345640296449 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361394355093234,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:47:29.264052827+08:00","start_ts":457114345640296449,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114345640296449,"checkpoint_ts":457114345640296449,"checkpoint_time":"2025-04-04 15:47:24.392"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x "is-owner": true, run task successfully + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore run task failed 1-th time, retry later Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 55250 0 --:--:-- --:--:-- --:--:-- 55250 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:47:24.392","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:47:30.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:24.392","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:30.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:24.392","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:30.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:47:24.392' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:24.392","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:30.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:24.392","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:30.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:47:30' + echo 'sink_checkpoint_ts is 2025-04-04' 15:47:24.392 sink_checkpoint_ts is 2025-04-04 15:47:24.392 ++ date -d '2025-04-04 15:47:24.392' +%s + checkpoint_timestamp=1743752844 ++ date -d '2025-04-04 15:47:30' +%s + current_timestamp=1743752850 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' owner pid: 5987 owner id 1a968a59-3033-4abb-ad18-a78918ce5cb6 [Fri Apr 4 15:47:30 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.60936095.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check_ts_forward 08831740-2868-468d-87ae-f758f41a6164 [2025/04/04 15:47:33.346 +08:00] [WARN] [diff.go:557] ["checksum is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] [sourceChecksum=155810305] [targetChecksum=2835274770] [getSourceChecksumCost=6.505963ms] [getTargetChecksumCost=2.227432ms] [2025/04/04 15:47:33.357 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 0, b: 0, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:25, }"] [2025/04/04 15:47:33.429 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 1, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:26, a: 1, }"] [2025/04/04 15:47:33.529 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:47:28, a: 3, b: 3, }"] [2025/04/04 15:47:33.629 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 6, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:31, a: 6, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361367627609027 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13bb14000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:9648, start at 2025-04-04 15:47:31.157514909 +0800 CST m=+20.173928402 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:31.164 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:31.141 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:31.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361367627609027 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13bb14000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:9648, start at 2025-04-04 15:47:31.157514909 +0800 CST m=+20.173928402 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:31.164 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:31.141 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:31.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361370927503246 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12bc30000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:9715, start at 2025-04-04 15:47:14.867214277 +0800 CST m=+3.807964831 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:14.874 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:14.878 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:14.878 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t1 exists + sleep 5 [2025/04/04 15:47:33.730 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 8, b: 8, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:33, }"] [2025/04/04 15:47:33.830 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 9, b: 9, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:34, }"] [2025/04/04 15:47:33.930 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 10, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:35, a: 10, }"] changefeed is working normally rts: 457114347657232393->457114347919376391 checkpoint: 457114347657232393->457114347919376391 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:47:30.281 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=08831740-2868-468d-87ae-f758f41a6164] [duration=36.045798ms] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION"] run task successfully [2025/04/04 15:47:34.030 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 11, b: 11, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:36, }"] [2025/04/04 15:47:34.130 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:47:37, a: 12, b: 12, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:33 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ac78c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114347854888964 MinTableBarrierTs:457114348117032967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/capture/73f32a01-0f60-4c0a-a68c-4584f71fb7ae {"id":"73f32a01-0f60-4c0a-a68c-4584f71fb7ae","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752851} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14610a5 73f32a01-0f60-4c0a-a68c-4584f71fb7ae /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114347854888964,"min-table-barrier-ts":457114348117032967,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/73f32a01-0f60-4c0a-a68c-4584f71fb7ae/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ac78c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114347854888964 MinTableBarrierTs:457114348117032967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/capture/73f32a01-0f60-4c0a-a68c-4584f71fb7ae {"id":"73f32a01-0f60-4c0a-a68c-4584f71fb7ae","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752851} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14610a5 73f32a01-0f60-4c0a-a68c-4584f71fb7ae /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114347854888964,"min-table-barrier-ts":457114348117032967,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/73f32a01-0f60-4c0a-a68c-4584f71fb7ae/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ac78c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114347854888964 MinTableBarrierTs:457114348117032967 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a968a59-3033-4abb-ad18-a78918ce5cb6 {"id":"1a968a59-3033-4abb-ad18-a78918ce5cb6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752845} /tidb/cdc/default/__cdc_meta__/capture/73f32a01-0f60-4c0a-a68c-4584f71fb7ae {"id":"73f32a01-0f60-4c0a-a68c-4584f71fb7ae","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752851} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461058 1a968a59-3033-4abb-ad18-a78918ce5cb6 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14610a5 73f32a01-0f60-4c0a-a68c-4584f71fb7ae /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114347854888964,"min-table-barrier-ts":457114348117032967,"admin-job-type":0} /tidb/cdc/default/default/task/position/1a968a59-3033-4abb-ad18-a78918ce5cb6/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/73f32a01-0f60-4c0a-a68c-4584f71fb7ae/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "1a968a59-3033-4abb-ad18-a78918ce5cb6" | grep id [2025/04/04 15:47:34.230 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 13, b: 13, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:38, }"] [2025/04/04 15:47:34.331 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 14, b: 14, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:39, }"] [2025/04/04 15:47:34.431 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:47:40, a: 15, b: 15, mST2i: s5yDgszOP7CkFnZdI, }"] wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361447656843048 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13cdd4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:36980, start at 2025-04-04 15:47:32.361084422 +0800 CST m=+2.392038944 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:32.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:32.341 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:32.341 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361447656843048 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13cdd4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:36980, start at 2025-04-04 15:47:32.361084422 +0800 CST m=+2.392038944 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:32.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:32.341 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:32.341 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361451751036162 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13cdcc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:37055, start at 2025-04-04 15:47:32.386034517 +0800 CST m=+2.332993119 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:32.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:32.390 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:32.390 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:47:34.531 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:47:42, a: 17, b: 17, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:34.631 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 18, b: 18, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:43, }"] [2025/04/04 15:47:34.731 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 19, b: 19, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:44, }"] [2025/04/04 15:47:34.832 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 20, b: 20, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:45, }"] [2025/04/04 15:47:34.932 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:47:46, a: 21, b: 21, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:35.032 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:47:47, a: 22, b: 22, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:35.132 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 23, b: 23, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:33, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:47:35.232 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 24, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:36, a: 24, }"] [2025/04/04 15:47:35.333 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 26, b: 26, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:51, }"] [2025/04/04 15:47:35.433 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:47:52, a: 27, b: 27, }"] "cluster-id": "default" "id": "73f32a01-0f60-4c0a-a68c-4584f71fb7ae", "cluster-id": "default" run task successfully wait process 9138 exit for 1-th time... wait process 9138 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9138) - No such process wait process 9138 exit for 3-th time... process 9138 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:47:35 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.92939295.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:47:35 CST 2025] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.2380323805.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:47:35.533 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 28, b: 28, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:53, }"] [2025/04/04 15:47:35.633 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:47:54, a: 29, b: 29, mST2i: s5yDgszOP7CkFnZdI, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:35 CST 2025] <<<<<< run test case ddl_reentrant success! >>>>>> [2025/04/04 15:47:35.733 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 30, b: 30, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:55, }"] [2025/04/04 15:47:35.833 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 32, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:57, a: 32, }"] [2025/04/04 15:47:35.934 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 33, b: 33, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:58, }"] [2025/04/04 15:47:36.034 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 34, b: 34, mST2i: s5yDgszOP7CkFnZdI, d: 15:47:59, }"] [2025/04/04 15:47:36.134 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 35, b: 35, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:00, }"] [2025/04/04 15:47:36.234 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 36, b: 36, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:01, }"] [2025/04/04 15:47:36.334 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 37, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:02, a: 37, }"] [2025/04/04 15:47:36.435 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:03, a: 38, b: 38, }"] The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls_with_user_auth Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:47:36.535 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 39, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:04, a: 39, }"] [2025/04/04 15:47:36.635 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 41, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:06, a: 41, }"] [2025/04/04 15:47:36.735 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 42, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:07, a: 42, }"] start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2025/04/04 15:47:36.835 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 44, b: 44, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:09, }"] [2025/04/04 15:47:36.936 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 45, b: 45, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:10, }"] [2025/04/04 15:47:37.036 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 47, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:12, a: 47, }"] [2025/04/04 15:47:37.136 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:13, a: 48, b: 48, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:37.236 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 49, b: 49, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:14, }"] [2025/04/04 15:47:37.336 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 50, b: 50, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:15, }"] [2025/04/04 15:47:37.436 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 51, b: 51, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:16, }"] [2025/04/04 15:47:37.537 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 52, b: 52, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:17, }"] [2025/04/04 15:47:37.637 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:18, a: 53, b: 53, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:37.737 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 54, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:19, a: 54, }"] Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later [2025/04/04 15:47:37.837 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:20, a: 55, b: 55, }"] [2025/04/04 15:47:37.937 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 56, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:21, a: 56, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 2-th check, retry later [2025/04/04 15:47:38.037 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:22, a: 57, b: 57, }"] [2025/04/04 15:47:38.138 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:23, a: 58, b: 58, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:38.238 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 59, b: 59, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:24, }"] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:38 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042955f0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336424624148 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46895fc1-e748-4ed4-b4ed-8321848afaf0 {"id":"46895fc1-e748-4ed4-b4ed-8321848afaf0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f59f 46895fc1-e748-4ed4-b4ed-8321848afaf0 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336424624148,"admin-job-type":0} /tidb/cdc/default/default/task/position/46895fc1-e748-4ed4-b4ed-8321848afaf0/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042955f0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336424624148 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46895fc1-e748-4ed4-b4ed-8321848afaf0 {"id":"46895fc1-e748-4ed4-b4ed-8321848afaf0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f59f 46895fc1-e748-4ed4-b4ed-8321848afaf0 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336424624148,"admin-job-type":0} /tidb/cdc/default/default/task/position/46895fc1-e748-4ed4-b4ed-8321848afaf0/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/59b51226-3245-4ea2-b1f4-5bdcb1533341 {UpstreamID:7489361195130315136 Namespace:default ID:59b51226-3245-4ea2-b1f4-5bdcb1533341 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:40.768591371 +0800 CST StartTs:457114334183292933 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042955f0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114334196137991} {CheckpointTs:457114336280182807 MinTableBarrierTs:457114336424624148 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114336280182807, checkpointTs: 457114336280182807, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46895fc1-e748-4ed4-b4ed-8321848afaf0 {"id":"46895fc1-e748-4ed4-b4ed-8321848afaf0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3f3f59f 46895fc1-e748-4ed4-b4ed-8321848afaf0 /tidb/cdc/default/default/changefeed/info/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"upstream-id":7489361195130315136,"namespace":"default","changefeed-id":"59b51226-3245-4ea2-b1f4-5bdcb1533341","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:40.768591371+08:00","start-ts":457114334183292933,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114334196137991} /tidb/cdc/default/default/changefeed/status/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":457114336280182807,"min-table-barrier-ts":457114336424624148,"admin-job-type":0} /tidb/cdc/default/default/task/position/46895fc1-e748-4ed4-b4ed-8321848afaf0/59b51226-3245-4ea2-b1f4-5bdcb1533341 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361195130315136 {"id":7489361195130315136,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table kill_owner_with_ddl.t1 exists table kill_owner_with_ddl.t2 exists table kill_owner_with_ddl.t3 not exists for 1-th check, retry later [2025/04/04 15:47:38.338 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:25, a: 60, b: 60, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:38.438 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 62, b: 62, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:27, }"] [2025/04/04 15:47:38.538 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 16:02:04, a: 63, b: 63, }"] [2025/04/04 15:47:38.639 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:29, a: 64, b: 64, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:38.739 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 67, b: 67, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:32, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:38 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ed66875-1090-4bd6-9d7a-37fe0cda9175 {"id":"7ed66875-1090-4bd6-9d7a-37fe0cda9175","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc489f605 7ed66875-1090-4bd6-9d7a-37fe0cda9175 /tidb/cdc/default/default/upstream/7489361367627609027 {"id":7489361367627609027,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ed66875-1090-4bd6-9d7a-37fe0cda9175 {"id":"7ed66875-1090-4bd6-9d7a-37fe0cda9175","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc489f605 7ed66875-1090-4bd6-9d7a-37fe0cda9175 /tidb/cdc/default/default/upstream/7489361367627609027 {"id":7489361367627609027,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ed66875-1090-4bd6-9d7a-37fe0cda9175 {"id":"7ed66875-1090-4bd6-9d7a-37fe0cda9175","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752855} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc489f605 7ed66875-1090-4bd6-9d7a-37fe0cda9175 /tidb/cdc/default/default/upstream/7489361367627609027 {"id":7489361367627609027,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.24082.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2025/04/04 15:47:38.839 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 68, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:33, a: 68, }"] [2025/04/04 15:47:38.939 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 69, b: 70, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:35, }"] ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 24300 0 --:--:-- --:--:-- --:--:-- 24300 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:47:37.743","puller_resolved_ts":"2025-04-04 15:47:30.742","last_synced_ts":"2025-04-04 15:47:31.242","now_ts":"2025-04-04 15:47:38.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:37.743","puller_resolved_ts":"2025-04-04' '15:47:30.742","last_synced_ts":"2025-04-04' '15:47:31.242","now_ts":"2025-04-04' '15:47:38.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:37.743","puller_resolved_ts":"2025-04-04' '15:47:30.742","last_synced_ts":"2025-04-04' '15:47:31.242","now_ts":"2025-04-04' '15:47:38.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 Create changefeed successfully! ID: 99052e56-4665-444c-86ea-c17a03e1f61d Info: {"upstream_id":7489361367627609027,"namespace":"default","id":"99052e56-4665-444c-86ea-c17a03e1f61d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:47:38.960275022+08:00","start_ts":457114349440860163,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114349440860163,"checkpoint_ts":457114349440860163,"checkpoint_time":"2025-04-04 15:47:38.890"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:47:39.039 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 69, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:58, a: 70, }"] [2025/04/04 15:47:39.140 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 71, b: 71, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:36, }"] [2025/04/04 15:47:39.240 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 72, b: 72, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:37, }"] nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later [2025/04/04 15:47:39.340 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 73, b: 73, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:38, }"] [2025/04/04 15:47:39.440 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:39, a: 74, b: 74, }"] [2025/04/04 15:47:39.540 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:41, a: 76, b: 76, }"] [2025/04/04 15:47:39.640 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 78, b: 78, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:46, }"] [2025/04/04 15:47:39.741 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:44, a: 79, b: 79, }"] Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:47:39.841 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:46, a: 81, b: 81, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:39.941 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 88, b: 88, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:53, }"] [2025/04/04 15:47:40.041 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 89, b: 89, mST2i: s5yDgszOP7CkFnZdI, d: 16:00:31, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 3-th check, retry later [2025/04/04 15:47:40.141 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:48:55, a: 90, b: 90, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:40.242 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:48:58, a: 93, b: 93, }"] + set +x [2025/04/04 15:47:40.342 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 94, mST2i: s5yDgszOP7CkFnZdI, d: 15:48:59, a: 94, }"] [2025/04/04 15:47:40.442 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 95, b: 95, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:00, }"] [2025/04/04 15:47:40.542 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 96, b: 96, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:47, }"] [2025/04/04 15:47:40.642 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 97, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:02, a: 97, }"] [2025/04/04 15:47:40.743 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 98, b: 98, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:03, }"] table kill_owner_with_ddl.t3 exists check diff failed 1-th time, retry later [2025/04/04 15:47:40.843 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 99, b: 99, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:04, }"] [2025/04/04 15:47:40.943 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 100, b: 100, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:05, }"] [2025/04/04 15:47:41.043 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 102, b: 102, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:07, }"] [2025/04/04 15:47:41.144 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 103, b: 103, mST2i: s5yDgszOP7CkFnZdI, d: 16:03:09, }"] [2025/04/04 15:47:41.244 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 104, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:22, a: 104, }"] [2025/04/04 15:47:41.344 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:10, a: 105, b: 105, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:41.444 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 106, b: 106, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:11, }"] [2025/04/04 15:47:41.544 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 108, b: 108, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:13, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361494586907635 TiDB Cluster ID. tikv_gc_leader_uuid 657ff145424000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:38582, start at 2025-04-04 15:47:40.96191862 +0800 CST m=+1.373869240 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:40.970 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:40.937 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:40.937 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:47:41.644 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:14, a: 109, b: 109, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:41.745 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 110, b: 110, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:15, }"] ***************** properties ***************** "mysql.port"="4000" "threadcount"="2" "requestdistribution"="uniform" "mysql.db"="multi_rocks" "readallfields"="true" "table"="a1" "updateproportion"="0" "dotransactions"="false" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "readproportion"="0" "recordcount"="1000" "insertproportion"="0" "scanproportion"="0" "workload"="core" ********************************************** [Fri Apr 4 15:47:41 CST 2025] <<<<<< START cdc server in http_api_tls_with_user_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls_with_user_auth.3863438636.out server --log-file /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_http_api_tls_with_user_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_data_http_api_tls_with_user_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls_with_user_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:47:41.845 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 111, b: 111, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:16, }"] [2025/04/04 15:47:41.945 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 112, b: 112, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:17, }"] [2025/04/04 15:47:42.045 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:20, a: 115, b: 115, }"] Run finished, takes 615.819897ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1718.4, Avg(us): 1197, Min(us): 817, Max(us): 33817, 95th(us): 2000, 99th(us): 2000 [2025/04/04 15:47:42.145 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 116, b: 116, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:21, }"] [2025/04/04 15:47:42.246 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 117, b: 117, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:22, }"] [2025/04/04 15:47:42.346 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 118, b: 118, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:23, }"] [2025/04/04 15:47:42.446 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 119, b: 119, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:24, }"] [2025/04/04 15:47:42.546 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:25, a: 120, b: 120, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361483326059007 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1449740002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:12155, start at 2025-04-04 15:47:40.256122764 +0800 CST m=+2.301392112 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:40.267 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:40.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:40.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361483326059007 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1449740002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:12155, start at 2025-04-04 15:47:40.256122764 +0800 CST m=+2.301392112 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:40.267 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:40.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:40.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361486025707605 TiDB Cluster ID. tikv_gc_leader_uuid 657ff144c2c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:12225, start at 2025-04-04 15:47:40.434133818 +0800 CST m=+2.363522494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:40.442 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:40.427 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:40.427 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_attributes.finish_mark not exists for 4-th check, retry later ***************** properties ***************** "requestdistribution"="uniform" "mysql.db"="multi_rocks" "operationcount"="0" "scanproportion"="0" "dotransactions"="false" "table"="a2" "readproportion"="0" "mysql.port"="4000" "updateproportion"="0" "recordcount"="1000" "mysql.user"="root" "insertproportion"="0" "mysql.host"="127.0.0.1" "threadcount"="2" "workload"="core" "readallfields"="true" ********************************************** check diff successfully [2025/04/04 15:47:42.646 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 121, b: 121, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:26, }"] [2025/04/04 15:47:42.747 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 122, b: 122, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:27, }"] [2025/04/04 15:47:42.847 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 123, b: 123, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:28, }"] [2025/04/04 15:47:42.947 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 124, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:29, a: 124, }"] [2025/04/04 15:47:43.047 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 125, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:30, a: 125, }"] check_redo_resolved_ts ec8b9d44-a91a-4b53-b4fc-f954b18f6437 457114350251409411 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/ec8b9d44-a91a-4b53-b4fc-f954b18f6437/meta global resolved ts 457114344772599810 not forward to 457114350251409411 run task failed 1-th time, retry later Run finished, takes 557.797291ms INSERT - Takes(s): 0.5, Count: 1000, OPS: 1855.1, Avg(us): 1084, Min(us): 765, Max(us): 18709, 95th(us): 2000, 99th(us): 2000 [2025/04/04 15:47:43.147 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 126, b: 126, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:31, }"] [2025/04/04 15:47:43.248 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:32, a: 127, b: 127, mST2i: s5yDgszOP7CkFnZdI, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:47:43.348 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 128, b: 128, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:33, }"] [2025/04/04 15:47:43.448 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 129, b: 129, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:34, }"] [2025/04/04 15:47:43.548 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:36, a: 131, b: 131, }"] nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later ***************** properties ***************** "workload"="core" "scanproportion"="0" "dotransactions"="false" "insertproportion"="0" "requestdistribution"="uniform" "recordcount"="1000" "operationcount"="0" "readallfields"="true" "threadcount"="2" "updateproportion"="0" "mysql.host"="127.0.0.1" "table"="a3" "mysql.port"="4000" "mysql.user"="root" "mysql.db"="multi_rocks" "readproportion"="0" ********************************************** [2025/04/04 15:47:43.648 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:37, a: 132, b: 132, }"] [2025/04/04 15:47:43.748 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 133, b: 133, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:38, }"] [2025/04/04 15:47:43.848 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:39, a: 134, b: 134, mST2i: s5yDgszOP7CkFnZdI, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:47:43.949 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:40, a: 135, b: 135, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:44.049 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 136, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:41, a: 136, }"] check_redo_resolved_ts ec8b9d44-a91a-4b53-b4fc-f954b18f6437 457114350251409411 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/ec8b9d44-a91a-4b53-b4fc-f954b18f6437/meta [2025/04/04 15:47:44.149 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 137, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:42, a: 137, }"] [2025/04/04 15:47:44.249 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 138, b: 138, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:43, }"] [2025/04/04 15:47:44.349 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 139, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:44, a: 139, }"] Run finished, takes 572.86117ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1810.2, Avg(us): 1116, Min(us): 783, Max(us): 20344, 95th(us): 2000, 99th(us): 2000 global resolved ts 457114344772599810 not forward to 457114350251409411 run task failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:44 CST 2025] <<<<<< run test case kill_owner_with_ddl success! >>>>>> + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13445.out cli tso query --pd=http://127.0.0.1:2379 table ddl_attributes.finish_mark not exists for 5-th check, retry later [2025/04/04 15:47:44.450 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 140, b: 140, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:45, }"] [2025/04/04 15:47:44.550 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 141, b: 141, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:46, }"] [2025/04/04 15:47:44.650 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 142, b: 142, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:47, }"] [2025/04/04 15:47:44.750 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:54, a: 143, b: 143, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:44.850 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 144, b: 144, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:49, }"] ***************** properties ***************** "readallfields"="true" "updateproportion"="0" "dotransactions"="false" "workload"="core" "mysql.user"="root" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "requestdistribution"="uniform" "operationcount"="0" "threadcount"="2" "readproportion"="0" "recordcount"="1000" "mysql.db"="multi_rocks" "table"="a4" "insertproportion"="0" ********************************************** [2025/04/04 15:47:44.950 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 145, b: 145, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:50, }"] [2025/04/04 15:47:45.050 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 146, b: 146, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:51, }"] + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:44 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:47:45.151 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 147, b: 147, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:52, }"] [2025/04/04 15:47:45.251 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:53, a: 148, b: 148, }"] [2025/04/04 15:47:45.351 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:55, a: 150, b: 150, }"] [2025/04/04 15:47:45.451 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 151, mST2i: s5yDgszOP7CkFnZdI, d: 15:49:56, a: 151, }"] [2025/04/04 15:47:45.551 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:49:57, a: 152, b: 152, }"] Run finished, takes 967.423451ms INSERT - Takes(s): 0.9, Count: 1000, OPS: 1053.8, Avg(us): 1898, Min(us): 843, Max(us): 76749, 95th(us): 2000, 99th(us): 19000 [2025/04/04 15:47:45.651 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 155, b: 155, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:00, }"] [2025/04/04 15:47:45.751 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:21, a: 156, b: 156, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:45.852 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 157, b: 157, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:02, }"] [2025/04/04 15:47:45.952 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 160, b: 160, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:05, }"] [2025/04/04 15:47:46.052 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 162, b: 162, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:07, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361498328782698 TiDB Cluster ID. tikv_gc_leader_uuid 657ff148400000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:23385, start at 2025-04-04 15:47:44.025768154 +0800 CST m=+2.155106126 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:44.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:44.000 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:44.000 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361498328782698 TiDB Cluster ID. tikv_gc_leader_uuid 657ff148400000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:23385, start at 2025-04-04 15:47:44.025768154 +0800 CST m=+2.155106126 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:44.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:44.000 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:44.000 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361492976955513 TiDB Cluster ID. tikv_gc_leader_uuid 657ff148590000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:23463, start at 2025-04-04 15:47:44.121800264 +0800 CST m=+2.176169804 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:44.131 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:44.100 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:44.100 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:47:46.152 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 163, b: 163, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:08, }"] [2025/04/04 15:47:46.252 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 166, b: 166, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:11, }"] [2025/04/04 15:47:46.352 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:12, a: 167, b: 167, }"] ***************** properties ***************** "insertproportion"="0" "mysql.port"="4000" "requestdistribution"="uniform" "readproportion"="0" "table"="a5" "recordcount"="1000" "mysql.host"="127.0.0.1" "dotransactions"="false" "threadcount"="2" "mysql.user"="root" "mysql.db"="multi_rocks" "workload"="core" "operationcount"="0" "updateproportion"="0" "readallfields"="true" "scanproportion"="0" ********************************************** + set +x + tso='457114350959460353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114350959460353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114350959460353 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:47:46 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1348013482.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_attributes.finish_mark not exists for 6-th check, retry later [2025/04/04 15:47:46.453 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 168, b: 168, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:06, }"] [2025/04/04 15:47:46.553 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 169, b: 169, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:14, }"] [2025/04/04 15:47:46.653 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 170, b: 170, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:15, }"] [2025/04/04 15:47:46.753 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 171, b: 171, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:16, }"] [2025/04/04 15:47:46.853 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 172, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:17, a: 172, }"] Run finished, takes 544.843339ms INSERT - Takes(s): 0.5, Count: 1000, OPS: 1902.5, Avg(us): 1054, Min(us): 763, Max(us): 19138, 95th(us): 2000, 99th(us): 2000 [Fri Apr 4 15:47:46 CST 2025] <<<<<< START cdc server in http_api_tls_with_user_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls_with_user_auth.3870938711.out server --log-file /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_http_api_tls_with_user_auth_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_data_http_api_tls_with_user_auth_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls_with_user_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:47:46.954 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 173, b: 173, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:18, }"] [2025/04/04 15:47:47.054 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 174, b: 174, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:19, }"] [2025/04/04 15:47:47.154 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 175, b: 175, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:20, }"] [2025/04/04 15:47:47.254 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 177, b: 177, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:22, }"] [2025/04/04 15:47:47.354 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 178, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:23, a: 178, }"] [2025/04/04 15:47:47.455 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 179, b: 179, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:24, }"] [2025/04/04 15:47:47.555 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:50:25, a: 180, b: 180, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:47.655 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 181, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:26, a: 181, }"] [2025/04/04 15:47:47.755 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 182, b: 182, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:27, }"] [2025/04/04 15:47:47.855 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:29, a: 184, b: 184, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:47:47.955 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 185, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:30, a: 185, }"] [2025/04/04 15:47:48.056 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:31, a: 186, b: 186, }"] [2025/04/04 15:47:48.156 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:00, a: 187, b: 187, mST2i: s5yDgszOP7CkFnZdI, }"] table ddl_attributes.finish_mark not exists for 7-th check, retry later [2025/04/04 15:47:48.256 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 188, b: 188, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:33, }"] [2025/04/04 15:47:48.356 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 189, b: 189, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:34, }"] check_redo_resolved_ts ec8b9d44-a91a-4b53-b4fc-f954b18f6437 457114350251409411 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/ec8b9d44-a91a-4b53-b4fc-f954b18f6437/meta [2025/04/04 15:47:48.456 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 190, b: 190, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:35, }"] [2025/04/04 15:47:48.557 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 191, b: 191, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:36, }"] [2025/04/04 15:47:48.657 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 192, b: 192, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:37, }"] run task successfully table multi_rocks.finish_mark not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.24790.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:47:48.757 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:38, a: 193, b: 193, }"] [2025/04/04 15:47:48.857 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 194, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:39, a: 194, }"] [2025/04/04 15:47:48.957 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 195, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:40, a: 195, }"] [2025/04/04 15:47:49.058 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 196, b: 196, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:55, }"] [2025/04/04 15:47:49.158 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 197, b: 197, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:42, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:47:49.258 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:50, a: 200, b: 200, }"] [2025/04/04 15:47:49.358 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 201, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:51, a: 201, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/682d0af2-d7dc-4095-9676-81e8e69c217d {"id":"682d0af2-d7dc-4095-9676-81e8e69c217d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752866} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4f367e4 682d0af2-d7dc-4095-9676-81e8e69c217d /tidb/cdc/default/default/upstream/7489361483326059007 {"id":7489361483326059007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/682d0af2-d7dc-4095-9676-81e8e69c217d {"id":"682d0af2-d7dc-4095-9676-81e8e69c217d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752866} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4f367e4 682d0af2-d7dc-4095-9676-81e8e69c217d /tidb/cdc/default/default/upstream/7489361483326059007 {"id":7489361483326059007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/682d0af2-d7dc-4095-9676-81e8e69c217d {"id":"682d0af2-d7dc-4095-9676-81e8e69c217d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752866} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4f367e4 682d0af2-d7dc-4095-9676-81e8e69c217d /tidb/cdc/default/default/upstream/7489361483326059007 {"id":7489361483326059007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114350959460353 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13531.out cli changefeed create --start-ts=457114350959460353 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361483326059007,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:47:49.580343271+08:00","start_ts":457114350959460353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114350959460353,"checkpoint_ts":457114350959460353,"checkpoint_time":"2025-04-04 15:47:44.683"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:47:49.458 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 202, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:52, a: 202, }"] [2025/04/04 15:47:49.559 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 203, b: 203, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:53, }"] [2025/04/04 15:47:49.659 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:55, a: 205, b: 205, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:47:49.759 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 206, b: 206, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:56, }"] [2025/04/04 15:47:49.859 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 207, b: 207, mST2i: s5yDgszOP7CkFnZdI, d: 15:50:57, }"] nonempty select id, val from test.availability1 where id=3 and val=3 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... + set +x + tso='457114351984967681 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114351984967681 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "workload"="core" "scanproportion"="0" "insertproportion"="0" "updateproportion"="0" "mysql.user"="root" "mysql.db"="move_table" "dotransactions"="false" "threadcount"="10" "readproportion"="0" "mysql.port"="4000" "requestdistribution"="uniform" "recordcount"="10000" "operationcount"="0" "mysql.host"="127.0.0.1" "readallfields"="true" ********************************************** table ddl_attributes.finish_mark exists [2025/04/04 15:47:49.959 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:58, a: 208, b: 208, }"] [2025/04/04 15:47:50.060 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:50:59, a: 209, b: 209, }"] [2025/04/04 15:47:50.160 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 210, b: 210, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:00, }"] + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:49 GMT < Content-Length: 1661 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/capture/ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a {"id":"ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752867} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503cd1c ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/capture/ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a {"id":"ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752867} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503cd1c ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d26411de-88f5-4619-894b-322ce459f9f4 {"id":"d26411de-88f5-4619-894b-322ce459f9f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752862} /tidb/cdc/default/__cdc_meta__/capture/ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a {"id":"ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752867} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503ccdd d26411de-88f5-4619-894b-322ce459f9f4 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc503cd1c ef0d89d3-37a9-49f7-a8e0-8e47327e8a7a /tidb/cdc/default/default/upstream/7489361494586907635 {"id":7489361494586907635,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x check diff successfully [2025/04/04 15:47:50.260 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 212, b: 212, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:02, }"] [2025/04/04 15:47:50.360 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:03, a: 213, b: 213, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_capture [Fri Apr 4 15:47:50 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.62816283.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_rocks.finish_mark not exists for 2-th check, retry later [2025/04/04 15:47:50.461 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 214, b: 214, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:04, }"] [2025/04/04 15:47:50.561 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 215, b: 215, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:05, }"] [2025/04/04 15:47:50.661 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:06, a: 216, b: 216, mST2i: s5yDgszOP7CkFnZdI, }"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2025/04/04 15:47:50.761 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 217, b: 217, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:07, }"] [2025/04/04 15:47:50.861 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 219, b: 219, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:09, }"] [2025/04/04 15:47:50.961 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 220, b: 220, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:10, }"] [2025/04/04 15:47:50.741 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:47:50.743 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:47:50.743 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:47:50.743 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:47:50.743 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114344785444893] [resolvedTs=457114351802777606] [2025/04/04 15:47:50.745 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.676784ms] [2025/04/04 15:47:50.746 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:47:50.746 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:47:50.746 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:47:50.753 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:47:50.756 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:47:50.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:47:50.757 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:47:50.758 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:47:50.758 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:47:50.765 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:47:50.768 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:47:50.768 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] wait process cdc.test exit for 1-th time... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' start tidb cluster in /tmp/tidb_cdc_test/multi_cdc_cluster Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:47:51.062 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 221, b: 221, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:11, }"] [2025/04/04 15:47:51.162 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 222, b: 222, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:12, }"] + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [2025/04/04 15:47:51.262 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:13, a: 223, b: 223, }"] [2025/04/04 15:47:51.362 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 224, b: 224, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:14, }"] [2025/04/04 15:47:51.462 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 226, b: 226, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:16, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:47:51.563 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:17, a: 227, b: 227, }"] [2025/04/04 15:47:51.663 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 228, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:18, a: 228, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2025/04/04 15:47:51.763 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 229, b: 229, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:19, }"] [2025/04/04 15:47:51.863 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 230, b: 230, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:21, }"] [2025/04/04 15:47:51.963 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:22, a: 231, b: 231, }"] [Fri Apr 4 15:47:52 CST 2025] <<<<<< run test case ddl_attributes success! >>>>>> pass test: check health [2025/04/04 15:47:52.063 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 232, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:23, a: 232, }"] [2025/04/04 15:47:52.164 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 233, b: 233, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:24, }"] pass test: get status [2025/04/04 15:47:52.264 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 234, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:25, a: 234, }"] [2025/04/04 15:47:52.364 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 235, b: 235, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:26, }"] [2025/04/04 15:47:52.464 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 236, b: 236, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:27, }"] table multi_rocks.finish_mark not exists for 3-th check, retry later Run finished, takes 2.14845759s INSERT - Takes(s): 2.1, Count: 10000, OPS: 4665.7, Avg(us): 2105, Min(us): 795, Max(us): 49897, 95th(us): 2000, 99th(us): 41000 [Fri Apr 4 15:47:52 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2487024872.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:47:52.565 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 237, b: 237, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:28, }"] [2025/04/04 15:47:52.665 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 238, b: 238, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:29, }"] [2025/04/04 15:47:52.812 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=2.06933829s] [2025/04/04 15:47:52.813 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114344785444888,\"CommitTs\":457114344785444893,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_file\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:47:52.765 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 239, b: 239, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:30, }"] [2025/04/04 15:47:52.865 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 240, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:31, a: 240, }"] [2025/04/04 15:47:52.965 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 241, b: 241, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:32, }"] table test.t1 exists + sleep 5 [2025/04/04 15:47:53.066 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:33, a: 242, b: 242, }"] [2025/04/04 15:47:53.166 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 244, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:35, a: 244, }"] pass test: create changefeed check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s [2025/04/04 15:47:53.266 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 245, b: 245, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:36, }"] [2025/04/04 15:47:53.367 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 246, b: 246, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:37, }"] [2025/04/04 15:47:53.467 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 247, b: 247, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:38, }"] + info='{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114353005232131, "checkpoint_time": "2025-04-04 15:47:52.487", "error": null }' + echo '{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114353005232131, "checkpoint_time": "2025-04-04 15:47:52.487", "error": null }' { "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114353005232131, "checkpoint_time": "2025-04-04 15:47:52.487", "error": null } ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114353005232131, '"checkpoint_time":' '"2025-04-04' '15:47:52.487",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114353005232131, '"checkpoint_time":' '"2025-04-04' '15:47:52.487",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:47:53.567 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 249, b: 249, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:40, }"] [2025/04/04 15:47:53.667 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 250, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:41, a: 250, }"] + info='{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114353044553745, "checkpoint_time": "2025-04-04 15:47:52.637", "error": null }' + echo '{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114353044553745, "checkpoint_time": "2025-04-04 15:47:52.637", "error": null }' { "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114353044553745, "checkpoint_time": "2025-04-04 15:47:52.637", "error": null } ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114353044553745, '"checkpoint_time":' '"2025-04-04' '15:47:52.637",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114353044553745, '"checkpoint_time":' '"2025-04-04' '15:47:52.637",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:53 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004486e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114353098031109 MinTableBarrierTs:457114353098031111 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114353098031109, checkpointTs: 457114353098031109, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88bada91-95ab-47ab-bba5-2f6345add44b {"id":"88bada91-95ab-47ab-bba5-2f6345add44b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752870} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461189 88bada91-95ab-47ab-bba5-2f6345add44b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114353098031109,"min-table-barrier-ts":457114353098031111,"admin-job-type":0} /tidb/cdc/default/default/task/position/88bada91-95ab-47ab-bba5-2f6345add44b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004486e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114353098031109 MinTableBarrierTs:457114353098031111 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114353098031109, checkpointTs: 457114353098031109, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88bada91-95ab-47ab-bba5-2f6345add44b {"id":"88bada91-95ab-47ab-bba5-2f6345add44b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752870} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461189 88bada91-95ab-47ab-bba5-2f6345add44b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114353098031109,"min-table-barrier-ts":457114353098031111,"admin-job-type":0} /tidb/cdc/default/default/task/position/88bada91-95ab-47ab-bba5-2f6345add44b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004486e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114353098031109 MinTableBarrierTs:457114353098031111 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114353098031109, checkpointTs: 457114353098031109, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114353098031111, checkpointTs: 457114353098031111, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88bada91-95ab-47ab-bba5-2f6345add44b {"id":"88bada91-95ab-47ab-bba5-2f6345add44b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752870} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461189 88bada91-95ab-47ab-bba5-2f6345add44b /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114353098031109,"min-table-barrier-ts":457114353098031111,"admin-job-type":0} /tidb/cdc/default/default/task/position/88bada91-95ab-47ab-bba5-2f6345add44b/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2025/04/04 15:47:53.768 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 251, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:42, a: 251, }"] [2025/04/04 15:47:53.868 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 252, b: 252, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:43, }"] [2025/04/04 15:47:53.968 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 253, b: 253, mST2i: s5yDgszOP7CkFnZdI, d: 16:01:02, }"] + info='{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114353096982531, "checkpoint_time": "2025-04-04 15:47:52.837", "error": null }' + echo '{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114353096982531, "checkpoint_time": "2025-04-04 15:47:52.837", "error": null }' { "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114353096982531, "checkpoint_time": "2025-04-04 15:47:52.837", "error": null } ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114353096982531, '"checkpoint_time":' '"2025-04-04' '15:47:52.837",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114353096982531, '"checkpoint_time":' '"2025-04-04' '15:47:52.837",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s [2025/04/04 15:47:54.167 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114351802777606] [2025/04/04 15:47:54.167 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114351802777606] [2025/04/04 15:47:54.068 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 254, b: 254, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:45, }"] [2025/04/04 15:47:54.168 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:46, a: 255, b: 255, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/owner_remove_table_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + info='{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114353136304132, "checkpoint_time": "2025-04-04 15:47:52.987", "error": null }' + echo '{ "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114353136304132, "checkpoint_time": "2025-04-04 15:47:52.987", "error": null }' { "upstream_id": 7489361494586907635, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114353136304132, "checkpoint_time": "2025-04-04 15:47:52.987", "error": null } ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114353136304132, '"checkpoint_time":' '"2025-04-04' '15:47:52.987",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361494586907635, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114353136304132, '"checkpoint_time":' '"2025-04-04' '15:47:52.987",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully [2025/04/04 15:47:54.268 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114351802777606] [2025/04/04 15:47:54.268 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114351802777606] [2025/04/04 15:47:54.269 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 256, b: 256, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:47, }"] [2025/04/04 15:47:54.369 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 257, b: 257, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:48, }"] [2025/04/04 15:47:54.469 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:51:50, a: 259, b: 259, }"] [2025/04/04 15:47:54.569 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 260, mST2i: s5yDgszOP7CkFnZdI, d: 16:02:05, a: 260, }"] [2025/04/04 15:47:54.669 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:52, a: 261, b: 261, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:54.770 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 262, b: 262, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:53, }"] [2025/04/04 15:47:54.870 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:54, a: 263, b: 263, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:54.970 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 264, b: 264, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:55, }"] [2025/04/04 15:47:54.970 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114351802777606] [2025/04/04 15:47:54.970 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114351802777606] table multi_rocks.finish_mark not exists for 4-th check, retry later [2025/04/04 15:47:55.070 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 265, b: 265, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:56, }"] [2025/04/04 15:47:55.170 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 266, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:57, a: 266, }"] [2025/04/04 15:47:55.270 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:58, a: 267, b: 267, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:55.071 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=457114351802777606] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:47:55.071 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] Apply redo log successfully "is-owner": true, run task successfully [2025/04/04 15:47:55.370 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 268, b: 268, mST2i: s5yDgszOP7CkFnZdI, d: 15:51:59, }"] [2025/04/04 15:47:55.471 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 269, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:00, a: 269, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:47:55.571 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 270, b: 270, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:01, }"] [2025/04/04 15:47:55.671 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 271, b: 271, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:02, }"] [2025/04/04 15:47:55.771 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 272, b: 272, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:03, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:55 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.24924.out cli changefeed create --start-ts=457114351984967681 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' Create changefeed successfully! ID: e3039190-cdcd-4026-9fa3-351c46e020da Info: {"upstream_id":7489361498328782698,"namespace":"default","id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:47:55.809930019+08:00","start_ts":457114351984967681,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114351984967681,"checkpoint_ts":457114351984967681,"checkpoint_time":"2025-04-04 15:47:48.595"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:47:55.871 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 273, b: 273, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:04, }"] [2025/04/04 15:47:55.972 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:05, a: 274, b: 274, mST2i: s5yDgszOP7CkFnZdI, }"] check diff successfully [Fri Apr 4 15:47:55 CST 2025] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> count(*) 5000 [2025/04/04 15:47:56.072 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:52:07, a: 276, b: 276, }"] [2025/04/04 15:47:56.172 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 277, b: 277, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:08, }"] [2025/04/04 15:47:56.272 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 278, b: 278, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:09, }"] [2025/04/04 15:47:56.372 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 280, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:11, a: 280, }"] [2025/04/04 15:47:56.473 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 281, b: 281, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:12, }"] owner pid: 6286 owner id 88bada91-95ab-47ab-bba5-2f6345add44b process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v '88bada91-95ab-47ab-bba5-2f6345add44b' /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461189 run task successfully process status: S nonempty select id, val from test.availability1 where id=2 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2025/04/04 15:47:56.573 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 282, b: 282, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:07, }"] [2025/04/04 15:47:56.673 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:14, a: 283, b: 283, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:56.773 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 284, b: 284, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:15, }"] table test.simple1 exists running list_changefeed table multi_rocks.finish_mark exists check diff successfully [2025/04/04 15:47:56.873 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:52:16, a: 285, b: 285, }"] [2025/04/04 15:47:56.974 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 286, b: 286, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:17, }"] pass test: list changefeed + set +x [Fri Apr 4 15:47:57 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2495324955.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:47:57.074 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 287, b: 287, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:18, }"] [2025/04/04 15:47:57.174 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:52:19, a: 288, b: 288, }"] [2025/04/04 15:47:57.274 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:52:20, a: 289, b: 289, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... [2025/04/04 15:47:57.374 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 290, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:23, a: 290, }"] [2025/04/04 15:47:57.475 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 291, b: 291, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:24, }"] start tidb cluster in /tmp/tidb_cdc_test/owner_remove_table_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... [2025/04/04 15:47:57.575 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:25, a: 292, b: 292, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:57.675 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 293, b: 293, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:26, }"] [2025/04/04 15:47:57.775 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 294, b: 294, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:27, }"] [2025/04/04 15:47:57.875 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:28, a: 295, b: 295, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:57.975 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 296, b: 296, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:29, }"] [2025/04/04 15:47:58.076 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:30, a: 297, b: 297, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:58.176 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 298, b: 298, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:31, }"] [2025/04/04 15:47:58.276 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 299, b: 299, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:32, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:58 CST 2025] <<<<<< run test case multi_rocks success! >>>>>> + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 11496 23.0 0.6 4871664 2503284 ? Sl 15:47 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 11497 23.6 0.6 4857332 2506460 ? Sl 15:47 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 11498 32.8 0.6 4876280 2542584 ? Sl 15:47 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 11500 31.3 0.6 4874740 2527592 ? Sl 15:47 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 17357 0 --:--:-- --:--:-- --:--:-- 17357 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:47:54.983","puller_resolved_ts":"2025-04-04 15:47:50.983","last_synced_ts":"2025-04-04 15:47:51.033","now_ts":"2025-04-04 15:47:58.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:54.983","puller_resolved_ts":"2025-04-04' '15:47:50.983","last_synced_ts":"2025-04-04' '15:47:51.033","now_ts":"2025-04-04' '15:47:58.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:54.983","puller_resolved_ts":"2025-04-04' '15:47:50.983","last_synced_ts":"2025-04-04' '15:47:51.033","now_ts":"2025-04-04' '15:47:58.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 running get_changefeed pass test: get changefeed [2025/04/04 15:47:58.376 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 301, b: 301, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:35, }"] [2025/04/04 15:47:58.476 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 302, b: 302, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:36, }"] [2025/04/04 15:47:58.576 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 305, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:39, a: 305, }"] [2025/04/04 15:47:58.677 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 306, b: 306, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:40, }"] [2025/04/04 15:47:58.777 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:41, a: 307, b: 307, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:58.877 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 308, b: 308, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:42, }"] [2025/04/04 15:47:58.977 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 309, b: 309, mST2i: s5yDgszOP7CkFnZdI, d: 16:01:43, }"] [2025/04/04 15:47:59.077 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 310, b: 310, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:44, }"] nonempty select id, val from test.availability1 where id=2 and val=22 run task successfully [2025/04/04 15:47:59.178 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 311, b: 311, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:45, }"] [2025/04/04 15:47:59.278 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 312, b: 312, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:46, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:47:59.378 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:47, a: 313, b: 313, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:59.478 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:48, a: 314, b: 314, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:47:59.578 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 315, b: 315, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:49, }"] [2025/04/04 15:47:59.679 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 316, b: 316, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:50, }"] [2025/04/04 15:47:59.779 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 317, b: 317, mST2i: s5yDgszOP7CkFnZdI, d: 16:04:00, }"] running pause_changefeed VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361561816485903 TiDB Cluster ID. tikv_gc_leader_uuid 657ff155494000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:10959, start at 2025-04-04 15:47:57.38525314 +0800 CST m=+2.155467532 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:57.393 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:57.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:57.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361561816485903 TiDB Cluster ID. tikv_gc_leader_uuid 657ff155494000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:10959, start at 2025-04-04 15:47:57.38525314 +0800 CST m=+2.155467532 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:57.393 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:57.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:57.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361561033539474 TiDB Cluster ID. tikv_gc_leader_uuid 657ff155490000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-r1g2f-25j5p-6x31k, pid:11029, start at 2025-04-04 15:47:57.382583396 +0800 CST m=+2.091919185 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:57.389 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:57.398 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:57.398 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... [2025/04/04 15:47:59.879 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 318, b: 318, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:52, }"] [2025/04/04 15:47:59.979 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 319, b: 319, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:53, }"] [2025/04/04 15:48:00.079 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 320, b: 320, mST2i: s5yDgszOP7CkFnZdI, d: 16:04:43, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit sql_check run sql_check, 127.0.0.1 [2025/04/04 15:48:00.180 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 321, b: 321, mST2i: s5yDgszOP7CkFnZdI, d: 15:52:57, }"] [2025/04/04 15:48:00.280 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 324, b: 324, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:00, }"] run task successfully run test case test_stop_processor [Fri Apr 4 15:48:00 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.64736475.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_stop_processor.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_stop_processor --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034ff290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352430874681 MinTableBarrierTs:457114354974720005 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352430874794,"min-table-barrier-ts":457114354974720005,"admin-job-type":0} /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034ff290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352430874681 MinTableBarrierTs:457114354974720005 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352430874794,"min-table-barrier-ts":457114354974720005,"admin-job-type":0} /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034ff290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352430874681 MinTableBarrierTs:457114354974720005 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352430874794,"min-table-barrier-ts":457114354974720005,"admin-job-type":0} /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:48:00 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2502825030.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:00.380 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:01, a: 325, b: 325, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:00.480 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 326, b: 326, mST2i: s5yDgszOP7CkFnZdI, d: 16:02:37, }"] [2025/04/04 15:48:00.581 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 327, b: 327, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:03, }"] [2025/04/04 15:48:00.681 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 328, b: 328, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:04, }"] [2025/04/04 15:48:00.781 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 329, b: 329, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:47, }"] [2025/04/04 15:48:00.881 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 331, b: 331, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:07, }"] [2025/04/04 15:48:00.981 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 332, b: 332, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:08, }"] [2025/04/04 15:48:01.082 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 333, b: 333, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:09, }"] pass test: pause changefeed [2025/04/04 15:48:01.182 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 334, b: 334, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:10, }"] [2025/04/04 15:48:01.282 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 335, b: 335, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:11, }"] [2025/04/04 15:48:01.382 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 336, b: 336, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:12, }"] [2025/04/04 15:48:01.482 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 337, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:13, a: 337, }"] [2025/04/04 15:48:01.582 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 338, b: 338, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:14, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12265.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:48:01.683 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:15, a: 339, b: 339, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:01.783 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 340, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:16, a: 340, }"] [2025/04/04 15:48:01.883 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 341, b: 341, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:17, }"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) running update_changefeed [2025/04/04 15:48:01.983 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 342, b: 342, mST2i: s5yDgszOP7CkFnZdI, d: 16:04:01, }"] [2025/04/04 15:48:02.083 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 343, b: 343, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:19, }"] [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } pass test: update changefeed [2025/04/04 15:48:02.184 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 344, b: 344, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:20, }"] [2025/04/04 15:48:02.284 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 345, b: 345, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:21, }"] [2025/04/04 15:48:02.384 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 346, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:22, a: 346, }"] [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [2025/04/04 15:48:02.484 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 347, b: 347, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:23, }"] [2025/04/04 15:48:02.584 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 348, b: 348, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:24, }"] [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [2025/04/04 15:48:02.685 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 349, b: 349, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:25, }"] [2025/04/04 15:48:02.785 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:26, a: 350, b: 350, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:02.885 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 351, b: 351, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:27, }"] [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [2025/04/04 15:48:02.985 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 352, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:28, a: 352, }"] [2025/04/04 15:48:03.085 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:53:29, a: 353, b: 353, }"] [2025/04/04 15:48:03.185 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 355, b: 355, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:31, }"] [2025/04/04 15:48:03.286 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 356, b: 356, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:32, }"] [2025/04/04 15:48:03.386 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:33, a: 357, b: 357, mST2i: s5yDgszOP7CkFnZdI, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2025/04/04 15:48:03.487 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 358, b: 358, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:34, }"] [2025/04/04 15:48:03.587 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 359, b: 359, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:35, }"] + set +x + tso='457114355499008001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114355499008001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f33d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114355719208967 MinTableBarrierTs:457114355732316162 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114355719208967, checkpointTs: 457114355719208967, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f {"id":"b543cc81-4f13-4b79-bf6c-9c138bcf9f4f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461202 b543cc81-4f13-4b79-bf6c-9c138bcf9f4f /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114355719208967,"min-table-barrier-ts":457114355732316162,"admin-job-type":0} /tidb/cdc/default/default/task/position/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f33d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114355719208967 MinTableBarrierTs:457114355732316162 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114355719208967, checkpointTs: 457114355719208967, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f {"id":"b543cc81-4f13-4b79-bf6c-9c138bcf9f4f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461202 b543cc81-4f13-4b79-bf6c-9c138bcf9f4f /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114355719208967,"min-table-barrier-ts":457114355732316162,"admin-job-type":0} /tidb/cdc/default/default/task/position/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e0d85ba6-6e52-4985-85ed-88baeff711a3 {UpstreamID:7489360440316725357 Namespace:default ID:e0d85ba6-6e52-4985-85ed-88baeff711a3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.670116938 +0800 CST StartTs:457114287679733762 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f33d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114289082204166} {CheckpointTs:457114355719208967 MinTableBarrierTs:457114355732316162 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114355719208967, checkpointTs: 457114355719208967, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114355732316162, checkpointTs: 457114355732316162, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f {"id":"b543cc81-4f13-4b79-bf6c-9c138bcf9f4f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1461202 b543cc81-4f13-4b79-bf6c-9c138bcf9f4f /tidb/cdc/default/default/changefeed/info/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"upstream-id":7489360440316725357,"namespace":"default","changefeed-id":"e0d85ba6-6e52-4985-85ed-88baeff711a3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.670116938+08:00","start-ts":457114287679733762,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114289082204166} /tidb/cdc/default/default/changefeed/status/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":457114355719208967,"min-table-barrier-ts":457114355732316162,"admin-job-type":0} /tidb/cdc/default/default/task/position/b543cc81-4f13-4b79-bf6c-9c138bcf9f4f/e0d85ba6-6e52-4985-85ed-88baeff711a3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360440316725357 {"id":7489360440316725357,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' running resume_changefeed + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001508240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352614375573 MinTableBarrierTs:457114355761152008 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47340cec-e7c8-43e4-b00f-6579b70711b3 {"id":"47340cec-e7c8-43e4-b00f-6579b70711b3","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752881} /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af9e 47340cec-e7c8-43e4-b00f-6579b70711b3 /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352614375573,"min-table-barrier-ts":457114355761152008,"admin-job-type":0} /tidb/cdc/default/default/task/position/47340cec-e7c8-43e4-b00f-6579b70711b3/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001508240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352614375573 MinTableBarrierTs:457114355761152008 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47340cec-e7c8-43e4-b00f-6579b70711b3 {"id":"47340cec-e7c8-43e4-b00f-6579b70711b3","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752881} /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af9e 47340cec-e7c8-43e4-b00f-6579b70711b3 /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352614375573,"min-table-barrier-ts":457114355761152008,"admin-job-type":0} /tidb/cdc/default/default/task/position/47340cec-e7c8-43e4-b00f-6579b70711b3/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e3039190-cdcd-4026-9fa3-351c46e020da {UpstreamID:7489361498328782698 Namespace:default ID:e3039190-cdcd-4026-9fa3-351c46e020da SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:47:55.809930019 +0800 CST StartTs:457114351984967681 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001508240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114353872666626} {CheckpointTs:457114352614375573 MinTableBarrierTs:457114355761152008 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47340cec-e7c8-43e4-b00f-6579b70711b3 {"id":"47340cec-e7c8-43e4-b00f-6579b70711b3","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752881} /tidb/cdc/default/__cdc_meta__/capture/7632c881-02c8-4b50-844c-b42610985c7a {"id":"7632c881-02c8-4b50-844c-b42610985c7a","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752877} /tidb/cdc/default/__cdc_meta__/capture/7f850f29-b6b2-48ba-b67a-9724cc293257 {"id":"7f850f29-b6b2-48ba-b67a-9724cc293257","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752872} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502aeff 7f850f29-b6b2-48ba-b67a-9724cc293257 /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af55 7632c881-02c8-4b50-844c-b42610985c7a /tidb/cdc/default/__cdc_meta__/owner/223195ffc502af9e 47340cec-e7c8-43e4-b00f-6579b70711b3 /tidb/cdc/default/default/changefeed/info/e3039190-cdcd-4026-9fa3-351c46e020da {"upstream-id":7489361498328782698,"namespace":"default","changefeed-id":"e3039190-cdcd-4026-9fa3-351c46e020da","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:47:55.809930019+08:00","start-ts":457114351984967681,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114353872666626} /tidb/cdc/default/default/changefeed/status/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":457114352614375573,"min-table-barrier-ts":457114355761152008,"admin-job-type":0} /tidb/cdc/default/default/task/position/47340cec-e7c8-43e4-b00f-6579b70711b3/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'etcd info' /tidb/cdc/default/default/task/position/7632c881-02c8-4b50-844c-b42610985c7a/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7f850f29-b6b2-48ba-b67a-9724cc293257/e3039190-cdcd-4026-9fa3-351c46e020da {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361498328782698 {"id":7489361498328782698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:03.687 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 360, b: 360, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:36, }"] [2025/04/04 15:48:03.787 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 361, b: 361, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:37, }"] [2025/04/04 15:48:03.888 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 362, b: 362, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:38, }"] pass test: resume changefeed table move_table.usertable exists go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/benbjohnson/clock v1.1.0 [2025/04/04 15:48:03.988 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 363, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:39, a: 363, }"] [2025/04/04 15:48:04.088 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 364, b: 364, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:40, }"] go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/gin-gonic/gin v1.9.1 [2025/04/04 15:48:04.188 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:41, a: 365, b: 365, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:04.288 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 366, b: 366, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:42, }"] [2025/04/04 15:48:04.388 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 367, b: 367, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:43, }"] go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/philhofer/fwd v1.1.1 [Fri Apr 4 15:48:04 CST 2025] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1230312305.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server1 --cluster-id test1 --addr 127.0.0.1:8300 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:04.489 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:44, a: 368, b: 368, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:04.589 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 369, b: 369, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:45, }"] go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 [2025/04/04 15:48:04.689 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:46, a: 370, b: 370, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:04.789 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 371, b: 371, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:47, }"] [2025/04/04 15:48:04.889 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:53:48, a: 372, b: 372, }"] go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/99designs/keyring v1.2.1 "is-owner": true, run task successfully [2025/04/04 15:48:04.989 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 373, b: 373, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:49, }"] [2025/04/04 15:48:05.090 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 374, b: 374, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:50, }"] go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/gorilla/websocket v1.5.1 running rebalance_table pass test: rebalance table [2025/04/04 15:48:05.190 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:51, a: 375, b: 375, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:05.290 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 376, b: 376, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:52, }"] [2025/04/04 15:48:05.390 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:53, a: 377, b: 377, mST2i: s5yDgszOP7CkFnZdI, }"] go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 [2025/04/04 15:48:05.490 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 378, b: 378, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:54, }"] [2025/04/04 15:48:05.590 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 379, b: 379, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:55, }"] [2025/04/04 15:48:05.691 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 380, b: 380, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:56, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:05.791 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:53:57, a: 381, b: 381, }"] [2025/04/04 15:48:05.891 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 382, b: 382, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:58, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:48:05.991 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 383, b: 383, mST2i: s5yDgszOP7CkFnZdI, d: 15:53:59, }"] [2025/04/04 15:48:06.091 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 384, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:00, a: 384, }"] [2025/04/04 15:48:06.192 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 385, b: 385, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:01, }"] running list_processor owner pid: 6478 owner id b543cc81-4f13-4b79-bf6c-9c138bcf9f4f [2025/04/04 15:48:06.292 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 386, b: 386, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:02, }"] [2025/04/04 15:48:06.392 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 387, b: 387, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:25, }"] pass test: list processors [2025/04/04 15:48:06.492 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 388, b: 388, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:04, }"] [2025/04/04 15:48:06.592 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 390, b: 390, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:09, }"] [2025/04/04 15:48:06.692 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:10, a: 391, b: 391, }"] [2025/04/04 15:48:06.793 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 392, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:11, a: 392, }"] [2025/04/04 15:48:06.893 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 393, b: 393, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:12, }"] [2025/04/04 15:48:06.993 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:13, a: 394, b: 394, }"] [2025/04/04 15:48:07.093 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:14, a: 395, b: 395, }"] [2025/04/04 15:48:07.193 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 396, b: 396, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:15, }"] running get_processor [2025/04/04 15:48:07.293 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 397, b: 397, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:16, }"] [2025/04/04 15:48:07.394 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 398, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:17, a: 398, }"] [2025/04/04 15:48:07.494 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:18, a: 399, b: 399, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:07.594 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 400, b: 400, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:19, }"] [2025/04/04 15:48:07.694 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 401, b: 401, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:20, }"] changefeed id: e0d85ba6-6e52-4985-85ed-88baeff711a3 curl -s -d "cf-id=e0d85ba6-6e52-4985-85ed-88baeff711a3&admin-job=1" http://127.0.0.1:8300/capture/owner/admin | grep true "status": true, + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:07 GMT < Content-Length: 802 < Content-Type: text/plain; charset=utf-8 < { [802 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/ded41306-1a3b-4a26-b659-10e063a2d5eb {"id":"ded41306-1a3b-4a26-b659-10e063a2d5eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752884} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffc53ac2ff ded41306-1a3b-4a26-b659-10e063a2d5eb /tidb/cdc/test1/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/ded41306-1a3b-4a26-b659-10e063a2d5eb {"id":"ded41306-1a3b-4a26-b659-10e063a2d5eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752884} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffc53ac2ff ded41306-1a3b-4a26-b659-10e063a2d5eb /tidb/cdc/test1/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/ded41306-1a3b-4a26-b659-10e063a2d5eb {"id":"ded41306-1a3b-4a26-b659-10e063a2d5eb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752884} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffc53ac2ff ded41306-1a3b-4a26-b659-10e063a2d5eb /tidb/cdc/test1/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:48:07 CST 2025] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1236212364.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server2 --cluster-id test2 --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:07.794 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 402, b: 402, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:21, }"] [2025/04/04 15:48:07.895 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 403, b: 403, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:22, }"] run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 89 100 35 100 54 648 1000 --:--:-- --:--:-- --:--:-- 1648 { "status": true, "message": "" }nonempty select id, val from test.availability1 where id=4 and val=4 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2025/04/04 15:48:07.995 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:23, a: 404, b: 404, }"] [2025/04/04 15:48:08.095 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 405, b: 405, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:24, }"] [2025/04/04 15:48:08.195 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:25, a: 406, b: 406, mST2i: s5yDgszOP7CkFnZdI, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ci_collation_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361593893219543 TiDB Cluster ID. tikv_gc_leader_uuid 657ff15dc9c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:10414, start at 2025-04-04 15:48:06.08265914 +0800 CST m=+2.494788547 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:06.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:06.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:06.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361593893219543 TiDB Cluster ID. tikv_gc_leader_uuid 657ff15dc9c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:10414, start at 2025-04-04 15:48:06.08265914 +0800 CST m=+2.494788547 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:06.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:06.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:06.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361589452543529 TiDB Cluster ID. tikv_gc_leader_uuid 657ff15df540009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7b293-7dhtl-cpsb1, pid:10485, start at 2025-04-04 15:48:06.247757076 +0800 CST m=+2.567742172 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:06.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:06.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:06.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:48:08.295 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 407, b: 407, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:26, }"] [2025/04/04 15:48:08.395 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 408, b: 408, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:27, }"] [2025/04/04 15:48:08.496 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 409, b: 409, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:28, }"] [2025/04/04 15:48:08.596 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:29, a: 410, b: 410, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:08.696 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:30, a: 411, b: 411, }"] [2025/04/04 15:48:08.796 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 412, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:31, a: 412, }"] [2025/04/04 15:48:08.896 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 413, b: 413, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:32, }"] [2025/04/04 15:48:08.996 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:33, a: 414, b: 414, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:09.097 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 415, b: 415, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:34, }"] [2025/04/04 15:48:09.197 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:35, a: 416, b: 416, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:09.297 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 417, b: 417, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:36, }"] [2025/04/04 15:48:09.397 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:37, a: 418, b: 418, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:09.497 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 419, b: 419, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:38, }"] go: downloading github.com/ardielle/ardielle-go v1.5.2 [2025/04/04 15:48:09.598 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 420, b: 420, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:39, }"] [2025/04/04 15:48:09.698 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 421, b: 421, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:40, }"] nonempty select id, val from test.availability1 where id=4 and val=4 pass test: get processor [2025/04/04 15:48:09.798 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 422, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:41, a: 422, }"] [2025/04/04 15:48:09.898 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 423, b: 423, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:42, }"] [2025/04/04 15:48:09.998 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 424, b: 424, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:43, }"] [Fri Apr 4 15:48:10 CST 2025] <<<<<< START cdc server in owner_remove_table_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/OwnerRemoveTableError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_remove_table_error.1177811780.out server --log-file /tmp/tidb_cdc_test/owner_remove_table_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_remove_table_error/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later [2025/04/04 15:48:10.099 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 426, b: 426, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:45, }"] [2025/04/04 15:48:10.199 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 427, b: 427, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:46, }"] start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:48:10.299 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:47, a: 428, b: 428, }"] [2025/04/04 15:48:10.399 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:54:48, a: 429, b: 429, }"] [2025/04/04 15:48:10.499 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 430, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:49, a: 430, }"] [2025/04/04 15:48:10.600 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 431, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:50, a: 431, }"] [2025/04/04 15:48:10.700 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 432, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:51, a: 432, }"] running move_table [2025/04/04 15:48:10.800 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:52, a: 433, b: 433, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:10.900 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:53, a: 434, b: 434, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:11.000 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 435, b: 435, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:54, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:10 GMT < Content-Length: 802 < Content-Type: text/plain; charset=utf-8 < { [802 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/64f290f4-a96f-4bcd-900b-bcedab007405 {"id":"64f290f4-a96f-4bcd-900b-bcedab007405","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffc53ac326 64f290f4-a96f-4bcd-900b-bcedab007405 /tidb/cdc/test2/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/64f290f4-a96f-4bcd-900b-bcedab007405 {"id":"64f290f4-a96f-4bcd-900b-bcedab007405","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffc53ac326 64f290f4-a96f-4bcd-900b-bcedab007405 /tidb/cdc/test2/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/64f290f4-a96f-4bcd-900b-bcedab007405 {"id":"64f290f4-a96f-4bcd-900b-bcedab007405","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffc53ac326 64f290f4-a96f-4bcd-900b-bcedab007405 /tidb/cdc/test2/default/upstream/7489361561816485903 {"id":7489361561816485903,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12429.out cli changefeed create --start-ts=457114355499008001 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed1.toml pass test: move table [2025/04/04 15:48:11.101 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 436, b: 436, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:55, }"] [2025/04/04 15:48:11.201 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 437, b: 437, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:56, }"] Create changefeed successfully! ID: f3ba7142-363a-4522-afbe-326552439851 Info: {"upstream_id":7489361561816485903,"namespace":"default","id":"f3ba7142-363a-4522-afbe-326552439851","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:48:11.122475132+08:00","start_ts":457114355499008001,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114355499008001,"checkpoint_ts":457114355499008001,"checkpoint_time":"2025-04-04 15:48:02.000"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/ci_collation_compatibility Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:48:11.301 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 438, b: 438, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:57, }"] [row2="{ mST2i: IsNull, d: 15:54:57, a: 438, b: 438, }"] [2025/04/04 15:48:11.401 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 439, b: 439, mST2i: s5yDgszOP7CkFnZdI, d: 15:54:58, }"] [row2="{ mST2i: IsNull, d: 15:54:58, a: 439, b: 439, }"] [2025/04/04 15:48:11.501 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 441, b: 441, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:00, }"] [2025/04/04 15:48:11.602 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 442, b: 442, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:01, }"] [2025/04/04 15:48:11.702 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:02, a: 443, b: 443, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:11.802 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 444, b: 444, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:03, }"] [row2="{ a: 444, b: 444, mST2i: IsNull, d: 15:55:03, }"] [2025/04/04 15:48:11.902 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 445, b: 445, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:04, }"] [row2="{ d: 15:55:04, a: 445, b: 445, mST2i: IsNull, }"] [2025/04/04 15:48:12.002 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 446, b: 446, mST2i: IsNull, d: 15:55:05, }"] Verifying downstream PD is started... [2025/04/04 15:48:12.102 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 447, b: 447, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:06, }"] [row2="{ mST2i: IsNull, d: 15:55:06, a: 447, b: 447, }"] [2025/04/04 15:48:12.203 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:55:07, a: 448, b: 448, }"] [row2="{ d: 15:55:07, a: 448, b: 448, mST2i: IsNull, }"] running set_log_level pass test: set log level [2025/04/04 15:48:12.303 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 449, b: 449, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:08, }"] [row2="{ a: 449, b: 449, mST2i: IsNull, d: 15:55:08, }"] [2025/04/04 15:48:12.403 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 450, b: 450, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:09, }"] [2025/04/04 15:48:12.503 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 451, b: 451, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:10, }"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12471.out cli changefeed create --start-ts=457114355499008001 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8301 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed2.toml [2025/04/04 15:48:12.603 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 452, b: 452, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:11, }"] [2025/04/04 15:48:12.704 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:12, a: 453, b: 453, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:12.804 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 454, b: 454, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:13, }"] [2025/04/04 15:48:12.904 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 455, b: 455, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:14, }"] [2025/04/04 15:48:13.004 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 456, b: 456, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:15, }"] [row2="{ a: 456, b: 456, mST2i: IsNull, d: 15:55:15, }"] Create changefeed successfully! ID: c3bdcd84-0474-42d5-89f9-6d462d685569 Info: {"upstream_id":7489361561816485903,"namespace":"default","id":"c3bdcd84-0474-42d5-89f9-6d462d685569","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:48:12.971326372+08:00","start_ts":457114355499008001,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc2"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114355499008001,"checkpoint_ts":457114355499008001,"checkpoint_time":"2025-04-04 15:48:02.000"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:48:13.104 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 457, b: 457, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:16, }"] [row2="{ a: 457, b: 457, mST2i: IsNull, d: 15:55:16, }"] [2025/04/04 15:48:13.204 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 458, b: 458, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:17, }"] [row2="{ d: 15:55:17, a: 458, b: 458, mST2i: IsNull, }"] [2025/04/04 15:48:13.305 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 459, b: 459, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:18, }"] [row2="{ a: 459, b: 459, mST2i: IsNull, d: 15:55:18, }"] Verifying downstream PD is started... running remove_changefeed + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:13 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef23358a-eb43-47e7-b389-342f69cfdb45 {"id":"ef23358a-eb43-47e7-b389-342f69cfdb45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752890} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5558ad6 ef23358a-eb43-47e7-b389-342f69cfdb45 /tidb/cdc/default/default/upstream/7489361593893219543 {"id":7489361593893219543,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef23358a-eb43-47e7-b389-342f69cfdb45 {"id":"ef23358a-eb43-47e7-b389-342f69cfdb45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752890} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5558ad6 ef23358a-eb43-47e7-b389-342f69cfdb45 /tidb/cdc/default/default/upstream/7489361593893219543 {"id":7489361593893219543,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef23358a-eb43-47e7-b389-342f69cfdb45 {"id":"ef23358a-eb43-47e7-b389-342f69cfdb45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752890} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5558ad6 ef23358a-eb43-47e7-b389-342f69cfdb45 /tidb/cdc/default/default/upstream/7489361593893219543 {"id":7489361593893219543,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:48:13.406 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 460, b: 460, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:19, }"] [2025/04/04 15:48:13.506 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:20, a: 461, b: 461, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:13.606 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:21, a: 462, b: 462, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:13.706 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 464, b: 464, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:23, }"] [2025/04/04 15:48:13.806 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 465, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:24, a: 465, }"] [row2="{ b: 465, mST2i: IsNull, d: 15:55:24, a: 465, }"] pass test: remove changefeed table owner_remove_table_error.finished_mark not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:48:13.906 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:55:25, a: 466, b: 466, }"] [row2="{ b: 466, mST2i: IsNull, d: 15:55:25, a: 466, }"] [2025/04/04 15:48:14.007 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 467, b: 467, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:26, }"] [row2="{ a: 467, b: 467, mST2i: IsNull, d: 15:55:26, }"] nonempty select id, val from test.availability1 where id=4 and val=4 run task successfully test_stop_processor pass [2025/04/04 15:48:14.107 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 468, b: 468, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:27, }"] [row2="{ a: 468, b: 468, mST2i: IsNull, d: 15:55:27, }"] [2025/04/04 15:48:14.207 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 469, b: 469, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:28, }"] [row2="{ a: 469, b: 469, mST2i: IsNull, d: 15:55:28, }"] [2025/04/04 15:48:14.307 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 470, b: 470, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:29, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:48:14.407 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 471, b: 471, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:30, }"] [2025/04/04 15:48:14.507 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:31, a: 472, b: 472, mST2i: s5yDgszOP7CkFnZdI, }"] + set +x table test.multi_cdc1 exists table test.multi_cdc2 exists [2025/04/04 15:48:14.607 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 473, b: 473, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:32, }"] [2025/04/04 15:48:14.708 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 474, b: 474, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:33, }"] [2025/04/04 15:48:14.808 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 475, mST2i: s5yDgszOP7CkFnZdI, d: 16:00:30, a: 475, }"] [row2="{ a: 475, b: 475, mST2i: IsNull, d: 15:55:34, }"] check diff failed 1-th time, retry later running resign_owner [2025/04/04 15:48:14.908 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 476, b: 476, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:35, }"] [row2="{ mST2i: IsNull, d: 15:55:35, a: 476, b: 476, }"] [2025/04/04 15:48:15.008 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 477, b: 477, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:36, }"] [row2="{ mST2i: IsNull, d: 15:55:36, a: 477, b: 477, }"] wait process cdc.test exit for 2-th time... pass test: resign owner Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:48:15.108 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 478, b: 478, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:37, }"] [row2="{ b: 478, mST2i: IsNull, d: 15:55:37, a: 478, }"] [2025/04/04 15:48:15.208 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 479, b: 479, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:38, }"] [row2="{ a: 479, b: 479, mST2i: IsNull, d: 15:55:38, }"] [2025/04/04 15:48:15.309 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 480, b: 480, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:39, }"] [2025/04/04 15:48:15.409 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 481, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:40, a: 481, }"] [2025/04/04 15:48:15.509 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 482, b: 482, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:41, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:15 CST 2025] <<<<<< run test case availability success! >>>>>> table owner_remove_table_error.finished_mark not exists for 2-th check, retry later [2025/04/04 15:48:15.609 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 483, b: 483, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:42, }"] [row2="{ b: 483, mST2i: IsNull, d: 15:55:42, a: 483, }"] [2025/04/04 15:48:15.709 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 484, b: 484, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:43, }"] [row2="{ a: 484, b: 484, mST2i: IsNull, d: 15:55:43, }"] [2025/04/04 15:48:15.810 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 485, b: 485, mST2i: IsNull, d: 15:55:44, }"] running get_tso [2025/04/04 15:48:15.910 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 486, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:45, a: 486, }"] [row2="{ a: 486, b: 486, mST2i: IsNull, d: 15:55:45, }"] [2025/04/04 15:48:16.010 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 487, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:46, a: 487, }"] [row2="{ a: 487, b: 487, mST2i: IsNull, d: 15:55:46, }"] [2025/04/04 15:48:16.110 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:55:47, a: 488, b: 488, }"] [row2="{ b: 488, mST2i: IsNull, d: 15:55:47, a: 488, }"] pass test: get tso [2025/04/04 15:48:16.210 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 489, b: 489, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:48, }"] [row2="{ mST2i: IsNull, d: 15:55:48, a: 489, b: 489, }"] [2025/04/04 15:48:16.311 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 490, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:50, a: 490, }"] [row2="{ a: 490, b: 490, mST2i: IsNull, d: 15:55:50, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:16.411 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 491, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:51, a: 491, }"] [row2="{ a: 491, b: 491, mST2i: IsNull, d: 15:55:51, }"] [2025/04/04 15:48:16.511 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 492, b: 492, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:52, }"] [row2="{ a: 492, b: 492, mST2i: IsNull, d: 15:55:52, }"] [2025/04/04 15:48:16.611 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 493, b: 493, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:53, }"] [row2="{ a: 493, b: 493, mST2i: IsNull, d: 15:55:53, }"] [2025/04/04 15:48:16.711 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ d: 15:55:54, a: 494, b: 494, mST2i: s5yDgszOP7CkFnZdI, }"] [row2="{ a: 494, b: 494, mST2i: IsNull, d: 15:55:54, }"] [2025/04/04 15:48:16.812 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 495, b: 495, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:55, }"] [row2="{ a: 495, b: 495, mST2i: IsNull, d: 15:55:55, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:16.912 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 496, b: 496, mST2i: IsNull, d: 15:55:56, }"] [2025/04/04 15:48:17.012 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 497, b: 497, mST2i: s5yDgszOP7CkFnZdI, d: 15:55:57, }"] [row2="{ a: 497, b: 497, mST2i: IsNull, d: 15:55:57, }"] [2025/04/04 15:48:17.112 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:55:58, a: 498, b: 498, }"] [row2="{ a: 498, b: 498, mST2i: IsNull, d: 15:55:58, }"] running verify_table check diff successfully wait process cdc.test exit for 1-th time... [2025/04/04 15:48:17.212 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:55:59, a: 499, b: 499, }"] [row2="{ a: 499, b: 499, mST2i: IsNull, d: 15:55:59, }"] [2025/04/04 15:48:17.313 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 510, b: 510, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:11, }"] [row2="{ a: 510, b: 510, mST2i: OAgIfmAaJaSuCkBgIEH8SXYxQLKQBAnnoycF, d: 15:56:11, }"] pass test: verify table [2025/04/04 15:48:17.413 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 511, b: 511, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:12, }"] [row2="{ d: 15:56:12, a: 511, b: 511, mST2i: meSVB4JgrEQJmP4gQFdf5, }"] [2025/04/04 15:48:17.513 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:56:13, a: 512, b: 512, }"] [row2="{ a: 512, b: 512, mST2i: uk4rIYKlIfVTqFQT8qr4UbIuItPnmLZsgEQ, d: 15:56:13, }"] [2025/04/04 15:48:17.613 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 513, b: 513, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:14, }"] [row2="{ d: 15:56:14, a: 513, b: 513, mST2i: KHhwYfbm3efOqJeQCa1h3ucNNP4FSHMviHwKUPItM7, }"] [2025/04/04 15:48:17.713 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 514, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:15, a: 514, }"] [row2="{ a: 514, b: 514, mST2i: orANtbu33x8ESHf6CDqJsqhJPdWULgkEQDkRBOeXA, d: 15:56:15, }"] [2025/04/04 15:48:17.814 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:56:16, a: 515, b: 515, }"] [row2="{ b: 515, mST2i: zOp7JdNxbNMNHpuROsu, d: 15:56:16, a: 515, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:48:17.914 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 516, b: 516, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:17, }"] [row2="{ a: 516, b: 516, mST2i: E1Xj5YkxrqtxS24RxtnSWaBbbcEHuHfIH0gs1, d: 15:56:17, }"] [2025/04/04 15:48:18.014 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 517, b: 517, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:18, }"] [row2="{ b: 517, mST2i: E8xc9Rpnm1HR3GJAdEBZ2o8vRMC7XCeMT, d: 15:56:18, a: 517, }"] [2025/04/04 15:48:18.114 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 518, b: 518, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:19, }"] [row2="{ b: 518, mST2i: fulRi7, d: 15:56:19, a: 518, }"] table owner_remove_table_error.finished_mark not exists for 3-th check, retry later [2025/04/04 15:48:18.214 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 519, b: 519, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:20, }"] [row2="{ a: 519, b: 519, mST2i: zvmNjZFXPjYnqPgmssN0FtV87a3kKEa, d: 15:56:20, }"] [2025/04/04 15:48:18.314 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 520, b: 520, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:21, }"] running create_changefeed_v2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:18 CST 2025] <<<<<< run test case multi_cdc_cluster success! >>>>>> [2025/04/04 15:48:18.415 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 521, b: 521, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:22, }"] [2025/04/04 15:48:18.515 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:23, a: 522, b: 522, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:18.615 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 523, b: 523, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:24, }"] [2025/04/04 15:48:18.715 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:56:25, a: 524, b: 524, }"] [2025/04/04 15:48:18.816 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:26, a: 525, b: 525, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:18.916 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:56:27, a: 526, b: 526, }"] [2025/04/04 15:48:19.016 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:28, a: 527, b: 527, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:19.116 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 528, b: 528, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:29, }"] pass test: create changefeed v2 [2025/04/04 15:48:19.216 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:30, a: 529, b: 529, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:19.316 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:56:31, a: 530, b: 530, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:19.416 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 531, b: 531, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:32, }"] [2025/04/04 15:48:19.516 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 532, b: 532, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:33, }"] [2025/04/04 15:48:19.617 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 533, b: 533, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:34, }"] [2025/04/04 15:48:19.717 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:35, a: 534, b: 534, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:19.817 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 535, b: 535, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:36, }"] [2025/04/04 15:48:19.917 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 536, b: 536, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:37, }"] [2025/04/04 15:48:20.017 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 537, b: 537, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:38, }"] [2025/04/04 15:48:20.118 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 538, b: 538, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:39, }"] table owner_remove_table_error.finished_mark exists check diff successfully running delete_changefeed_v2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361635576940779 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16a408000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:12837, start at 2025-04-04 15:48:18.835469233 +0800 CST m=+2.712157276 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:18.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:18.818 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:18.818 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361635576940779 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16a408000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:12837, start at 2025-04-04 15:48:18.835469233 +0800 CST m=+2.712157276 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:18.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:18.818 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:18.818 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361643620048135 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16a6280007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:12915, start at 2025-04-04 15:48:18.963595884 +0800 CST m=+2.737239541 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:18.970 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:19.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:19.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:48:20.218 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 539, b: 539, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:40, }"] [2025/04/04 15:48:20.318 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 540, b: 540, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:41, }"] [2025/04/04 15:48:20.418 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 541, b: 541, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:42, }"] wait process cdc.test exit for 1-th time... Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:48:20.518 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:43, a: 542, b: 542, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:20.619 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 543, b: 543, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:44, }"] pass test: remove changefeed v2 [2025/04/04 15:48:20.719 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 544, b: 544, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:45, }"] [2025/04/04 15:48:20.819 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 546, b: 546, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:47, }"] [2025/04/04 15:48:20.919 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:48, a: 547, b: 547, mST2i: s5yDgszOP7CkFnZdI, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:48:21.019 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:49, a: 548, b: 548, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:21.120 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 549, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:50, a: 549, }"] [2025/04/04 15:48:21.220 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 550, b: 550, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:51, }"] [2025/04/04 15:48:21.320 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:52, a: 551, b: 551, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:21.420 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 552, b: 552, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:53, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361640090581239 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16a900000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:28014, start at 2025-04-04 15:48:19.153516774 +0800 CST m=+2.136283299 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:19.161 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:19.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:19.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361640090581239 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16a900000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:28014, start at 2025-04-04 15:48:19.153516774 +0800 CST m=+2.136283299 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:19.161 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:19.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:19.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361648463519507 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16aa900009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-k1gsf-t0xw1-6rmjd, pid:28084, start at 2025-04-04 15:48:19.25080346 +0800 CST m=+2.155990884 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:19.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:19.236 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:19.236 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:21 CST 2025] <<<<<< run test case owner_remove_table_error success! >>>>>> [2025/04/04 15:48:21.520 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 553, b: 553, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:54, }"] [2025/04/04 15:48:21.620 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 554, b: 554, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:55, }"] running unsafe_apis [2025/04/04 15:48:21.721 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 555, b: 555, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:56, }"] [2025/04/04 15:48:21.821 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:57, a: 556, b: 556, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:21.921 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 557, b: 557, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:58, }"] status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock [2025/04/04 15:48:22.021 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 558, b: 558, mST2i: s5yDgszOP7CkFnZdI, d: 15:56:59, }"] [2025/04/04 15:48:22.121 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:57:00, a: 559, b: 559, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:22.222 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 560, b: 560, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:02, }"] [2025/04/04 15:48:22.322 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 561, b: 561, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:03, }"] [2025/04/04 15:48:22.422 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 562, b: 562, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:04, }"] [2025/04/04 15:48:22.522 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:05, a: 563, b: 563, }"] [2025/04/04 15:48:22.622 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 564, b: 564, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:06, }"] [2025/04/04 15:48:22.723 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 565, b: 565, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:07, }"] [2025/04/04 15:48:22.823 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 566, b: 566, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:08, }"] [2025/04/04 15:48:22.923 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 567, b: 567, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:09, }"] [Fri Apr 4 15:48:22 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1417114173.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:23.023 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 568, b: 568, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:10, }"] [2025/04/04 15:48:23.123 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 569, b: 569, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:11, }"] [2025/04/04 15:48:23.224 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: 0NiEUHudOGlR4lcOakldj0ITMQ3AK6qzbQfsYaU, d: 15:57:12, a: 570, b: 570, }"] [2025/04/04 15:48:23.324 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:57:13, a: 571, b: 571, mST2i: ArgkXHrKq, }"] [2025/04/04 15:48:23.425 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 572, b: 572, mST2i: T9sobKm9PG, d: 15:57:14, }"] [2025/04/04 15:48:23.525 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: bn3fBrIvOdW878jYxUZeIJQhYewem797e5, d: 15:57:15, a: 573, b: 573, }"] [2025/04/04 15:48:23.625 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 574, b: 574, mST2i: hA5d2pYvEK98ptFfjmrduq7fmn, d: 15:57:16, }"] [2025/04/04 15:48:23.725 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 575, b: 575, mST2i: mPGJvmaj8DMeNP, d: 15:57:17, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:48:23.825 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 576, b: 576, mST2i: PBw5aT, d: 15:57:18, }"] [2025/04/04 15:48:23.925 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 577, b: 577, mST2i: vzaSWi, d: 15:57:19, }"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.cli.29389.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... [2025/04/04 15:48:24.026 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 578, b: 578, mST2i: xTmGKBpOqwQUyn9ofoQxc57hmUjQcNWFHu, d: 15:57:20, }"] [2025/04/04 15:48:24.126 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 579, b: 579, mST2i: FV6eMeAvnbJW1xugEAgVztZ9uHqICVG2u1yRU8o, d: 15:57:21, }"] [2025/04/04 15:48:24.226 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 580, b: 580, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:22, }"] [2025/04/04 15:48:24.326 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:23, a: 581, b: 581, }"] [2025/04/04 15:48:24.426 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 582, b: 582, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:24, }"] wait process cdc.test exit for 3-th time... [2025/04/04 15:48:24.527 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:57:25, a: 583, b: 583, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:24.627 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 584, b: 584, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:26, }"] [2025/04/04 15:48:24.727 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 585, b: 585, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:27, }"] [2025/04/04 15:48:24.827 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 586, b: 586, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:28, }"] [2025/04/04 15:48:24.927 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 587, b: 587, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:29, }"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:48:25 CST 2025] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> [2025/04/04 15:48:25.027 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:30, a: 588, b: 588, }"] [2025/04/04 15:48:25.128 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 589, b: 589, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:31, }"] [2025/04/04 15:48:25.228 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ d: 15:57:32, a: 590, b: 590, mST2i: s5yDgszOP7CkFnZdI, }"] [row2="{ b: 590, mST2i: 3gg0ehiS3ia6JCsryC1uMebQCz784o, d: 15:57:32, a: 590, }"] + set +x + tso='457114361197232129 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114361197232129 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:48:25 CST 2025] <<<<<< START cdc server in ci_collation_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.2942729429.out server --log-file /tmp/tidb_cdc_test/ci_collation_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ci_collation_compatibility/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:25.328 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 591, b: 591, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:33, }"] [row2="{ a: 591, b: 591, mST2i: MiCecmGdVAS2GVg2iQ, d: 15:57:33, }"] [2025/04/04 15:48:25.428 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 592, b: 592, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:34, }"] [row2="{ d: 15:57:34, a: 592, b: 592, mST2i: dmC7S1y, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + CDC_BINARY=cdc.test + SINK_TYPE=mysql + TEST_HOST_LIST=(UP_TIDB_HOST UP_PD_HOST_{1..3} UP_TIKV_HOST_{1..3}) + lan_addrs=($(hostname -I)) ++ hostname -I + lan_addr=10.233.93.110 + export UP_TIDB_HOST=10.233.93.110 UP_PD_HOST_1=10.233.93.110 UP_PD_HOST_2=10.233.93.110 UP_PD_HOST_3=10.233.93.110 UP_TIKV_HOST_1=10.233.93.110 UP_TIKV_HOST_2=10.233.93.110 UP_TIKV_HOST_3=10.233.93.110 + UP_TIDB_HOST=10.233.93.110 + UP_PD_HOST_1=10.233.93.110 + UP_PD_HOST_2=10.233.93.110 + UP_PD_HOST_3=10.233.93.110 + UP_TIKV_HOST_1=10.233.93.110 + UP_TIKV_HOST_2=10.233.93.110 + UP_TIKV_HOST_3=10.233.93.110 + proxy_pid= ++ shuf -i 10081-20081 -n1 + proxy_port=19398 + trap 'stop_tidb_cluster && stop_proxy' EXIT + '[' mysql == mysql ']' + prepare + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIDB_HOST + echo 'UP_TIDB_HOST 10.233.93.110' UP_TIDB_HOST 10.233.93.110 + case $(printenv $host) in ++ printenv UP_TIDB_HOST + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_1 + echo 'UP_PD_HOST_1 10.233.93.110' UP_PD_HOST_1 10.233.93.110 + case $(printenv $host) in ++ printenv UP_PD_HOST_1 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_2 + echo 'UP_PD_HOST_2 10.233.93.110' UP_PD_HOST_2 10.233.93.110 + case $(printenv $host) in ++ printenv UP_PD_HOST_2 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_3 + echo 'UP_PD_HOST_3 10.233.93.110' UP_PD_HOST_3 10.233.93.110 + case $(printenv $host) in ++ printenv UP_PD_HOST_3 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_1 + echo 'UP_TIKV_HOST_1 10.233.93.110' UP_TIKV_HOST_1 10.233.93.110 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_1 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_2 + echo 'UP_TIKV_HOST_2 10.233.93.110' UP_TIKV_HOST_2 10.233.93.110 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_2 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_3 + echo 'UP_TIKV_HOST_3 10.233.93.110' UP_TIKV_HOST_3 10.233.93.110 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_3 + rm -rf /tmp/tidb_cdc_test/http_proxies + mkdir -p /tmp/tidb_cdc_test/http_proxies + stop_tidb_cluster [2025/04/04 15:48:25.528 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 593, b: 593, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:35, }"] [row2="{ a: 593, b: 593, mST2i: 7bL6IrK3Syiu6jlkpUglH01g3d17NLVfY, d: 15:57:35, }"] [2025/04/04 15:48:25.629 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 594, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:36, a: 594, }"] [row2="{ b: 594, mST2i: lW7rK28W6JP8JAZi2UZqIIZ3GwzJMrC, d: 15:57:36, a: 594, }"] [2025/04/04 15:48:25.729 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 595, b: 595, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:37, }"] [row2="{ d: 15:57:37, a: 595, b: 595, mST2i: uSvL4laZsgJT6i0V2xxv2AAh0mgw2pjftetDzWh4lKBFL, }"] [2025/04/04 15:48:25.829 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 596, b: 596, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:38, }"] [row2="{ mST2i: 3lEvi, d: 15:57:38, a: 596, b: 596, }"] [2025/04/04 15:48:25.929 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:39, a: 597, b: 597, }"] [row2="{ a: 597, b: 597, mST2i: 5spqL4lXWIzSyX7Gd8IQ2EN5PApNeXILX, d: 15:57:39, }"] [2025/04/04 15:48:26.029 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 598, b: 598, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:40, }"] [row2="{ a: 598, b: 598, mST2i: CJV, d: 15:57:40, }"] [2025/04/04 15:48:26.129 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 599, b: 599, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:41, }"] [row2="{ b: 599, mST2i: f0M4IeL5unGLjmJfvcRuSCwhv15c, d: 15:57:41, a: 599, }"] [2025/04/04 15:48:26.230 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 600, b: 600, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:43, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:25 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bd926043-22b1-4f08-815f-d6cf674b9331 {"id":"bd926043-22b1-4f08-815f-d6cf674b9331","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752902} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58635dd bd926043-22b1-4f08-815f-d6cf674b9331 /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bd926043-22b1-4f08-815f-d6cf674b9331 {"id":"bd926043-22b1-4f08-815f-d6cf674b9331","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752902} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58635dd bd926043-22b1-4f08-815f-d6cf674b9331 /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bd926043-22b1-4f08-815f-d6cf674b9331 {"id":"bd926043-22b1-4f08-815f-d6cf674b9331","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752902} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58635dd bd926043-22b1-4f08-815f-d6cf674b9331 /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "recordcount"="100" "mysql.user"="root" "fieldcount"="1000" "requestdistribution"="uniform" "scanproportion"="0" "operationcount"="0" "mysql.db"="consistent_replicate_storage_file_large_value" "dotransactions"="false" "updateproportion"="0" "fieldlength"="15000" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "readallfields"="true" "workload"="core" "threadcount"="10" "fieldlengthdistribution"="constant" "insertproportion"="0" ********************************************** [2025/04/04 15:48:26.330 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 601, b: 601, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:44, }"] [2025/04/04 15:48:26.430 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 602, b: 602, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:45, }"] [2025/04/04 15:48:26.530 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:57:46, a: 603, b: 603, mST2i: s5yDgszOP7CkFnZdI, }"] [2025/04/04 15:48:26.630 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:47, a: 604, b: 604, }"] [2025/04/04 15:48:26.730 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:48, a: 605, b: 605, }"] [2025/04/04 15:48:26.831 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:49, a: 606, b: 606, }"] [2025/04/04 15:48:26.931 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 607, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:50, a: 607, }"] [2025/04/04 15:48:27.031 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 608, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:51, a: 608, }"] [2025/04/04 15:48:27.131 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 609, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:52, a: 609, }"] [2025/04/04 15:48:27.231 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ mST2i: s5yDgszOP7CkFnZdI, d: 15:57:53, a: 610, b: 610, }"] [row2="{ a: 610, b: 610, mST2i: 4nzZbHc7bNdhKdGxIO8Ax15uwM2, d: 15:57:53, }"] [2025/04/04 15:48:27.331 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 611, b: 611, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:54, }"] [row2="{ a: 611, b: 611, mST2i: g8xhkvVvoRhkPFL9hIG79qiWkB1hS8, d: 15:57:54, }"] [2025/04/04 15:48:27.432 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 612, b: 612, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:55, }"] [row2="{ a: 612, b: 612, mST2i: CbaVfPJORhMzkglBrJ00lftcMDw, d: 15:57:55, }"] [2025/04/04 15:48:27.532 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 613, b: 613, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:56, }"] [row2="{ b: 613, mST2i: ccRjQznEQOT8WkY2UvYKdJu55, d: 15:57:56, a: 613, }"] [2025/04/04 15:48:27.257 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:48:27.258 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:48:27.259 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=7f850f29-b6b2-48ba-b67a-9724cc293257] [2025/04/04 15:48:27.259 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:48:27.259 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:48:27.389 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=47340cec-e7c8-43e4-b00f-6579b70711b3] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:48:27.539 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=7632c881-02c8-4b50-844c-b42610985c7a] [processorDetail="{\"table_ids\":[116]}"] [2025/04/04 15:48:27.688 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=7f850f29-b6b2-48ba-b67a-9724cc293257] [processorDetail="{\"table_ids\":[114]}"] [2025/04/04 15:48:27.688 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"47340cec-e7c8-43e4-b00f-6579b70711b3\":[],\"7632c881-02c8-4b50-844c-b42610985c7a\":[{\"ID\":116,\"Changefeed\":\"e3039190-cdcd-4026-9fa3-351c46e020da\"}],\"7f850f29-b6b2-48ba-b67a-9724cc293257\":[{\"ID\":114,\"Changefeed\":\"e3039190-cdcd-4026-9fa3-351c46e020da\"}]}"] [2025/04/04 15:48:27.688 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=e3039190-cdcd-4026-9fa3-351c46e020da&target-cp-id=7f850f29-b6b2-48ba-b67a-9724cc293257&table-id=116"] [2025/04/04 15:48:27.739 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:48:27.739 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=7632c881-02c8-4b50-844c-b42610985c7a] [targetCapture=7f850f29-b6b2-48ba-b67a-9724cc293257] [2025/04/04 15:48:27.632 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ d: 15:57:57, a: 614, b: 614, mST2i: d7ThyzsrG00lvRsXMc6MCoNTObVjb6qS66gIUT, }"] [2025/04/04 15:48:27.732 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 615, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:14, a: 615, }"] [row2="{ a: 615, b: 615, mST2i: 5Gks9hjHDMkzfjswKPy5nhk, d: 15:57:58, }"] table move_table.check1 exists [2025/04/04 15:48:27.832 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 616, b: 616, mST2i: s5yDgszOP7CkFnZdI, d: 15:57:59, }"] [row2="{ a: 616, b: 616, mST2i: G9, d: 15:57:59, }"] [2025/04/04 15:48:27.932 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ b: 617, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:00, a: 617, }"] [row2="{ d: 15:58:00, a: 617, b: 617, mST2i: 4aOU, }"] [2025/04/04 15:48:28.032 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 618, b: 618, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:01, }"] [row2="{ d: 15:58:01, a: 618, b: 618, mST2i: MCMJ, }"] check diff successfully [2025/04/04 15:48:28.133 +08:00] [WARN] [diff.go:937] ["find different row"] [column=mST2i] [row1="{ a: 619, b: 619, mST2i: s5yDgszOP7CkFnZdI, d: 15:58:02, }"] [row2="{ a: 619, b: 619, mST2i: d, d: 15:58:02, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:28 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e7945f2-6c34-48b8-b403-bb5c6a414c61 {"id":"9e7945f2-6c34-48b8-b403-bb5c6a414c61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752905} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58bc2eb 9e7945f2-6c34-48b8-b403-bb5c6a414c61 /tidb/cdc/default/default/upstream/7489361640090581239 {"id":7489361640090581239,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e7945f2-6c34-48b8-b403-bb5c6a414c61 {"id":"9e7945f2-6c34-48b8-b403-bb5c6a414c61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752905} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58bc2eb 9e7945f2-6c34-48b8-b403-bb5c6a414c61 /tidb/cdc/default/default/upstream/7489361640090581239 {"id":7489361640090581239,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e7945f2-6c34-48b8-b403-bb5c6a414c61 {"id":"9e7945f2-6c34-48b8-b403-bb5c6a414c61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752905} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc58bc2eb 9e7945f2-6c34-48b8-b403-bb5c6a414c61 /tidb/cdc/default/default/upstream/7489361640090581239 {"id":7489361640090581239,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.cli.29489.out cli changefeed create --start-ts=457114361197232129 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?safe-mode=true' + start_tidb_cluster --workdir /tmp/tidb_cdc_test/http_proxies The 1 times to try to start tidb cluster... Create changefeed successfully! ID: 104e302b-5ffe-4938-9342-7b6efde413ac Info: {"upstream_id":7489361640090581239,"namespace":"default","id":"104e302b-5ffe-4938-9342-7b6efde413ac","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:48:28.657960776+08:00","start_ts":457114361197232129,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114361197232129,"checkpoint_ts":457114361197232129,"checkpoint_time":"2025-04-04 15:48:23.737"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // stage [Pipeline] // cache [Pipeline] } [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials start tidb cluster in /tmp/tidb_cdc_test/http_proxies Starting Upstream PD... [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:48:32.349 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:48:32.351 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:48:32.351 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=7f850f29-b6b2-48ba-b67a-9724cc293257] [2025/04/04 15:48:32.352 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:48:32.352 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:48:32.488 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=47340cec-e7c8-43e4-b00f-6579b70711b3] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:48:32.638 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=7632c881-02c8-4b50-844c-b42610985c7a] [processorDetail="{\"table_ids\":[]}"] [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:48:32.788 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=e3039190-cdcd-4026-9fa3-351c46e020da] [captureID=7f850f29-b6b2-48ba-b67a-9724cc293257] [processorDetail="{\"table_ids\":[116,118]}"] [2025/04/04 15:48:32.788 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"47340cec-e7c8-43e4-b00f-6579b70711b3\":[],\"7632c881-02c8-4b50-844c-b42610985c7a\":[],\"7f850f29-b6b2-48ba-b67a-9724cc293257\":[{\"ID\":116,\"Changefeed\":\"e3039190-cdcd-4026-9fa3-351c46e020da\"},{\"ID\":118,\"Changefeed\":\"e3039190-cdcd-4026-9fa3-351c46e020da\"}]}"] [2025/04/04 15:48:32.789 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=e3039190-cdcd-4026-9fa3-351c46e020da&target-cp-id=7632c881-02c8-4b50-844c-b42610985c7a&table-id=116"] [2025/04/04 15:48:32.839 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:48:32.839 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=e3039190-cdcd-4026-9fa3-351c46e020da&target-cp-id=7632c881-02c8-4b50-844c-b42610985c7a&table-id=118"] [2025/04/04 15:48:32.889 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=118] [2025/04/04 15:48:32.889 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=7f850f29-b6b2-48ba-b67a-9724cc293257] [targetCapture=7632c881-02c8-4b50-844c-b42610985c7a] [Pipeline] // stage [Pipeline] } check diff successfully table move_table.check2 not exists for 1-th check, retry later Verifying downstream PD is started... table move_table.check2 exists check diff successfully wait process cdc.test exit for 1-th time... Run finished, takes 8.778963135s INSERT_ERROR - Takes(s): 8.0, Count: 100, OPS: 12.6, Avg(us): 258479, Min(us): 119547, Max(us): 399576, 95th(us): 375000, 99th(us): 387000 table consistent_replicate_storage_file_large_value.usertable exists table consistent_replicate_storage_file_large_value.check1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:36 CST 2025] <<<<<< run test case move_table success! >>>>>> ____________________________________ *************************** 1. row *************************** primary_ts: 457114347110400000 secondary_ts: 457114347160469548 primary_ts is 457114347110400000 start_ts is 457114345247604737 check successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:30 CST 2025] <<<<<< run test case syncpoint_check_ts success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_file_large_value.check1 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.93.110' (111) [2025/04/04 15:48:31.677 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] [2025/04/04 15:48:31.917 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:48:31.925 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:48:32.113 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:48:32.115 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:48:32.127 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:48:32.128 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:48:33.340 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:48:33.356 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:48:33.561 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:48:33.565 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:48:33.569 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:48:33.572 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:48:33.769 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:48:33.781 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:48:35.040 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:48:35.046 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:48:35.048 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:48:35.048 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:48:35.285 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:48:35.288 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:48:36.635 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:48:36.641 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:48:36.645 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:48:36.648 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] table consistent_replicate_storage_file_large_value.check1 exists table consistent_replicate_storage_file_large_value.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361733787761564 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17f430000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:7762, start at 2025-04-04 15:48:40.375348757 +0800 CST m=+2.035613191 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:40.382 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:40.381 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:40.381 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.93.110' (111) start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table consistent_replicate_storage_file_large_value.t2 not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:48:42.190 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] [2025/04/04 15:48:42.446 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:48:42.453 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:48:42.666 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:48:42.668 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:48:42.672 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:48:42.673 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:48:42.871 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361733787761564 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17f430000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:7762, start at 2025-04-04 15:48:40.375348757 +0800 CST m=+2.035613191 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:40.382 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:40.381 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:40.381 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361742383455086 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17f8140002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:7847, start at 2025-04-04 15:48:40.582247657 +0800 CST m=+2.141539414 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:40.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:40.581 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:40.581 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_proxies/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_proxies/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["10.233.93.110:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:48:42.889 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:48:43.080 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:48:43.083 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:48:43.100 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:48:43.104 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:48:43.298 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:48:43.325 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] start tidb cluster in /tmp/tidb_cdc_test/server_config_compatibility Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:48:43.529 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:48:43.531 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:48:43.562 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:48:43.570 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] table consistent_replicate_storage_file_large_value.t2 not exists for 3-th check, retry later [2025/04/04 15:48:43.740 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:48:43.776 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:48:43.961 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:48:43.963 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:48:43.982 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:48:43.990 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:48:44.159 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:48:44.182 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:48:44.375 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:48:44.377 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:48:44.392 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:48:44.399 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:48:44.582 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:48:44.600 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] + start_proxy + echo 'dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log...' dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log... + proxy_pid=9089 + echo 'proxy port: 19398' proxy port: 19398 + echo 'proxy pid: 9089' proxy pid: 9089 + sleep 5 + GO111MODULE=on + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + go run /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go --port=19398 go: downloading go1.23.4 (linux/amd64) [2025/04/04 15:48:44.784 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:48:44.786 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:48:44.809 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:48:44.820 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:44.978 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:48:45.014 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:48:45.189 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:48:45.191 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:48:45.237 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:48:45.248 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] Verifying downstream PD is started... [2025/04/04 15:48:45.395 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:48:45.439 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:48:45.595 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:48:45.597 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:48:45.651 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:48:45.663 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] table consistent_replicate_storage_file_large_value.t2 not exists for 4-th check, retry later [2025/04/04 15:48:45.803 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:48:45.866 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:48:46.013 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:48:46.015 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:48:46.081 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:48:46.092 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:48:46.222 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:48:46.300 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:48:46.436 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:48:46.439 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:48:46.521 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:48:46.532 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:48:46.640 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:48:46.731 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:48:46.844 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:48:46.846 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:48:46.940 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:48:46.950 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:48:47.045 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:48:47.154 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table consistent_replicate_storage_file_large_value.t2 not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361771939810038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff185d48000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:40253, start at 2025-04-04 15:48:47.08349123 +0800 CST m=+2.557769213 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:47.091 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:47.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:47.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361771939810038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff185d48000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:40253, start at 2025-04-04 15:48:47.08349123 +0800 CST m=+2.557769213 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:47.091 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:47.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:47.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361769807106774 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1858940009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:40320, start at 2025-04-04 15:48:46.771888398 +0800 CST m=+2.183204086 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:46.782 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:46.825 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:46.825 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 not exists for 6-th check, retry later go: downloading google.golang.org/grpc v1.64.0 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/pkg/errors v0.9.1 go: downloading golang.org/x/net v0.33.0 go: downloading golang.org/x/sys v0.28.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading github.com/gorilla/websocket v1.5.1 + export http_proxy=http://127.0.0.1:19398 + http_proxy=http://127.0.0.1:19398 + export https_proxy=http://127.0.0.1:19398 + https_proxy=http://127.0.0.1:19398 + echo 'try to connect pd cluster via proxy, pd addr: 10.233.93.110:2379' try to connect pd cluster via proxy, pd addr: 10.233.93.110:2379 + ensure 10 curl http://10.233.93.110:2379/ curl http://10.233.93.110:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 19398: Connection refused run task failed 1-th time, retry later go: downloading golang.org/x/text v0.21.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading google.golang.org/protobuf v1.35.2 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 exists The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 curl http://10.233.93.110:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 19398: Connection refused run task failed 2-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:53 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1444314445.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361781580222706 TiDB Cluster ID. tikv_gc_leader_uuid 657ff189ecc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:14963, start at 2025-04-04 15:48:51.284391017 +0800 CST m=+2.084574439 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:51.329 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:51.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:51.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361781580222706 TiDB Cluster ID. tikv_gc_leader_uuid 657ff189ecc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:14963, start at 2025-04-04 15:48:51.284391017 +0800 CST m=+2.084574439 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:51.329 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:51.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:51.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361791151326897 TiDB Cluster ID. tikv_gc_leader_uuid 657ff18a128000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:15049, start at 2025-04-04 15:48:51.448221245 +0800 CST m=+2.176154021 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:51.457 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:51.452 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:51.452 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:48:53.069 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2025/04/04 15:48:53.290 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:48:53.291 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:48:53.586 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:48:53.586 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:48:53.588 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:48:53.588 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:48:53.845 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:48:53.849 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:48:54.031 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:48:54.033 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:48:54.040 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:48:54.042 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:48:54.285 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:48:54.303 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:48:54.488 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:48:54.490 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:48:54.512 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:48:54.514 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:48:54.683 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:48:54.705 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:48:54.937 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:48:54.939 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:48:54.958 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:48:54.961 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:48:55.138 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:55.159 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:48:55.346 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:48:55.349 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:48:55.356 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:48:55.358 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.16255.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:48:55.731 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:48:55.737 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:48:55.943 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:48:55.945 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:48:55.948 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:48:55.950 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] curl http://10.233.93.110:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 19398: Connection refused run task failed 3-th time, retry later Starting TLS TiDB... [2025/04/04 15:48:56.142 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:48:56.145 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:56.339 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:48:56.341 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:48:56.346 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:48:56.348 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:56 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {UpstreamID:7489361635576940779 Namespace:default ID:a3a0ae83-847b-4d93-8b79-bd65d93fe849 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:48:25.904608046 +0800 CST StartTs:457114361729908745 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004129e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114361756123142} {CheckpointTs:457114369450049540 MinTableBarrierTs:457114369450049541 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/77d90fdc-a2c2-476f-bbfb-51db755ae239 {"id":"77d90fdc-a2c2-476f-bbfb-51db755ae239","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752933} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5863743 77d90fdc-a2c2-476f-bbfb-51db755ae239 /tidb/cdc/default/default/changefeed/info/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"upstream-id":7489361635576940779,"namespace":"default","changefeed-id":"a3a0ae83-847b-4d93-8b79-bd65d93fe849","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:48:25.904608046+08:00","start-ts":457114361729908745,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114361756123142} /tidb/cdc/default/default/changefeed/status/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":457114369450049540,"min-table-barrier-ts":457114369450049541,"admin-job-type":0} /tidb/cdc/default/default/task/position/77d90fdc-a2c2-476f-bbfb-51db755ae239/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {UpstreamID:7489361635576940779 Namespace:default ID:a3a0ae83-847b-4d93-8b79-bd65d93fe849 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:48:25.904608046 +0800 CST StartTs:457114361729908745 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004129e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114361756123142} {CheckpointTs:457114369450049540 MinTableBarrierTs:457114369450049541 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/77d90fdc-a2c2-476f-bbfb-51db755ae239 {"id":"77d90fdc-a2c2-476f-bbfb-51db755ae239","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752933} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5863743 77d90fdc-a2c2-476f-bbfb-51db755ae239 /tidb/cdc/default/default/changefeed/info/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"upstream-id":7489361635576940779,"namespace":"default","changefeed-id":"a3a0ae83-847b-4d93-8b79-bd65d93fe849","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:48:25.904608046+08:00","start-ts":457114361729908745,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114361756123142} /tidb/cdc/default/default/changefeed/status/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":457114369450049540,"min-table-barrier-ts":457114369450049541,"admin-job-type":0} /tidb/cdc/default/default/task/position/77d90fdc-a2c2-476f-bbfb-51db755ae239/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {UpstreamID:7489361635576940779 Namespace:default ID:a3a0ae83-847b-4d93-8b79-bd65d93fe849 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:48:25.904608046 +0800 CST StartTs:457114361729908745 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004129e60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114361756123142} {CheckpointTs:457114369450049540 MinTableBarrierTs:457114369450049541 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114369450049540, checkpointTs: 457114369450049540, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/77d90fdc-a2c2-476f-bbfb-51db755ae239 {"id":"77d90fdc-a2c2-476f-bbfb-51db755ae239","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752933} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5863743 77d90fdc-a2c2-476f-bbfb-51db755ae239 /tidb/cdc/default/default/changefeed/info/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"upstream-id":7489361635576940779,"namespace":"default","changefeed-id":"a3a0ae83-847b-4d93-8b79-bd65d93fe849","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:48:25.904608046+08:00","start-ts":457114361729908745,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114361756123142} /tidb/cdc/default/default/changefeed/status/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":457114369450049540,"min-table-barrier-ts":457114369450049541,"admin-job-type":0} /tidb/cdc/default/default/task/position/77d90fdc-a2c2-476f-bbfb-51db755ae239/a3a0ae83-847b-4d93-8b79-bd65d93fe849 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361635576940779 {"id":7489361635576940779,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:48:56.531 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:48:56.542 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:48:56.725 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:48:56.726 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:48:56.739 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:48:56.741 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:48:56.919 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:48:56.935 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:48:57.107 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:48:57.108 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:48:57.136 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:48:57.138 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:48:57.304 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:48:57.341 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] + set +x + tso='457114369628569601 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114369628569601 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:48:57 CST 2025] <<<<<< START cdc server in server_config_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.1630816310.out server --log-file /tmp/tidb_cdc_test/server_config_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/server_config_compatibility/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/server_config_compatibility/conf/server.toml ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:57.667 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:48:57.669 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:48:57.712 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:48:57.714 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:48:57.867 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:48:57.977 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361821928155017 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1901bc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:41828, start at 2025-04-04 15:48:57.628837975 +0800 CST m=+1.431763799 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:57.636 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:57.634 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:57.634 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + pd_host=127.0.0.1 + pd_port=2579 + is_tls=true + '[' true == true ']' ++ run_cdc_cli tso query --pd=https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41882.out cli tso query --pd=https://127.0.0.1:2579 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361815798487038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff18fdc80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:30109, start at 2025-04-04 15:48:57.343317786 +0800 CST m=+2.124578084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:57.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:57.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:57.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361815798487038 TiDB Cluster ID. tikv_gc_leader_uuid 657ff18fdc80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:30109, start at 2025-04-04 15:48:57.343317786 +0800 CST m=+2.124578084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:57.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:57.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:57.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361818587670588 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1900dc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:30191, start at 2025-04-04 15:48:57.534096955 +0800 CST m=+2.218465093 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:57.541 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:57.527 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:57.527 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114370358378497 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114370358378497 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:49:00 CST 2025] <<<<<< START cdc server in cli_tls_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.4191441916.out server --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_cli_tls_with_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_data_cli_tls_with_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/cli_tls_with_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:00 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69896481-eb09-440f-b77f-c37a4b7e28fd {"id":"69896481-eb09-440f-b77f-c37a4b7e28fd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752937} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc60999fc 69896481-eb09-440f-b77f-c37a4b7e28fd /tidb/cdc/default/default/upstream/7489361781580222706 {"id":7489361781580222706,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69896481-eb09-440f-b77f-c37a4b7e28fd {"id":"69896481-eb09-440f-b77f-c37a4b7e28fd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752937} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc60999fc 69896481-eb09-440f-b77f-c37a4b7e28fd /tidb/cdc/default/default/upstream/7489361781580222706 {"id":7489361781580222706,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69896481-eb09-440f-b77f-c37a4b7e28fd {"id":"69896481-eb09-440f-b77f-c37a4b7e28fd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752937} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc60999fc 69896481-eb09-440f-b77f-c37a4b7e28fd /tidb/cdc/default/default/upstream/7489361781580222706 {"id":7489361781580222706,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.16363.out cli changefeed create --start-ts=457114369628569601 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 5c3f402e-6c71-4249-abab-45d981e48aa6 Info: {"upstream_id":7489361781580222706,"namespace":"default","id":"5c3f402e-6c71-4249-abab-45d981e48aa6","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:00.871147436+08:00","start_ts":457114369628569601,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114369628569601,"checkpoint_ts":457114369628569601,"checkpoint_time":"2025-04-04 15:48:55.900"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:49:01 CST 2025] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.3142831430.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ci_collation_compatibility.t exists table ci_collation_compatibility.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:01 CST 2025] <<<<<< run test case ci_collation_compatibility success! >>>>>> + set +x curl http://10.233.93.110:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 19398: Connection refused run task failed 4-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later time="2025-04-04T15:49:02+08:00" level=info msg="Listening on 127.0.0.1:19398" time="2025-04-04T15:49:02+08:00" level=info msg="Not intercepting TLS connections" + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:03 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 {"id":"06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752940} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc626fdf2 06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 /tidb/cdc/default/default/upstream/7489361821928155017 {"id":7489361821928155017,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 {"id":"06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752940} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc626fdf2 06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 /tidb/cdc/default/default/upstream/7489361821928155017 {"id":7489361821928155017,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 {"id":"06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752940} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc626fdf2 06df0bdc-0ab7-4bc5-9154-69d4f7ea25f2 /tidb/cdc/default/default/upstream/7489361821928155017 {"id":7489361821928155017,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:49:03.943 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2025/04/04 15:49:04.152 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:49:04.154 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:49:04.347 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:49:04.350 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:49:04.356 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:49:04.358 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:49:04.548 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > [2025/04/04 15:49:04.572 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:49:04.753 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:49:04.755 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:49:04.770 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:49:04.772 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:04 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aaee3fa9-52e2-438b-a232-225ab452e3c2 {"id":"aaee3fa9-52e2-438b-a232-225ab452e3c2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752941} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6231bd5 aaee3fa9-52e2-438b-a232-225ab452e3c2 /tidb/cdc/default/default/upstream/7489361815798487038 {"id":7489361815798487038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aaee3fa9-52e2-438b-a232-225ab452e3c2 {"id":"aaee3fa9-52e2-438b-a232-225ab452e3c2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752941} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6231bd5 aaee3fa9-52e2-438b-a232-225ab452e3c2 /tidb/cdc/default/default/upstream/7489361815798487038 {"id":7489361815798487038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aaee3fa9-52e2-438b-a232-225ab452e3c2 {"id":"aaee3fa9-52e2-438b-a232-225ab452e3c2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752941} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6231bd5 aaee3fa9-52e2-438b-a232-225ab452e3c2 /tidb/cdc/default/default/upstream/7489361815798487038 {"id":7489361815798487038,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 852ad539-4ce0-4d29-82a8-a064d9e409a4 Info: {"upstream_id":7489361815798487038,"namespace":"default","id":"852ad539-4ce0-4d29-82a8-a064d9e409a4","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:04.843905193+08:00","start_ts":457114371956408322,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114371956408322,"checkpoint_ts":457114371956408322,"checkpoint_time":"2025-04-04 15:49:04.780"} table autorandom_test.table_a not exists for 1-th check, retry later [2025/04/04 15:49:04.948 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:49:04.963 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:49:05.145 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:49:05.147 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:49:05.159 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:49:05.161 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41986.out cli changefeed create --start-ts=457114370358378497 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [2025/04/04 15:49:05.339 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:49:05.345 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:49:05.532 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:49:05.535 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:49:05.535 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:49:05.537 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [WARN] --tz is deprecated in changefeed settings. [2025/04/04 15:49:05.738 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:49:05.744 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361821928155017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:05.864262803+08:00","start_ts":457114370358378497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114370358378497,"checkpoint_ts":457114370358378497,"checkpoint_time":"2025-04-04 15:48:58.684"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:49:05.931 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:49:05.933 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:49:05.944 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:49:05.946 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:49:06.121 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:49:06.144 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:49:06.326 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:49:06.328 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:49:06.341 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:49:06.343 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:49:06.517 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:49:06.526 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:49:06.705 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:49:06.707 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:49:06.720 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:49:06.722 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:49:05.206 +08:00] [WARN] [diff.go:766] ["rows is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] [cost=1m31.859648762s] [2025/04/04 15:49:05.208 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] check data successfully [2025/04/04 15:49:06.903 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:49:06.908 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:49:07.090 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:49:07.092 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:49:07.099 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:49:07.101 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] wait process cdc.test exit for 1-th time... [2025/04/04 15:49:07.278 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:49:07.298 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] + set +x table test.simple not exists for 1-th check, retry later table autorandom_test.table_a exists check diff failed 1-th time, retry later [2025/04/04 15:49:07.452 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:49:07.453 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:49:07.492 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:49:07.494 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:49:07.807 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:49:07.854 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] wait process cdc.test exit for 2-th time... [2025/04/04 15:49:08.008 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:49:08.011 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:49:08.082 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:49:08.085 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:08 CST 2025] <<<<<< run test case server_config_compatibility success! >>>>>> [2025/04/04 15:49:08.428 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:49:08.459 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:49:08.630 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:49:08.632 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:49:08.659 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:49:08.662 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:49:08.834 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] check diff successfully wait process cdc.test exit for 1-th time... table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... curl http://10.233.93.110:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 19000 0 --:--:-- --:--:-- --:--:-- 19000 404 page not found run task successfully + echo started proxy pid: 9089 started proxy pid: 9089 + echo started proxy at port: 19398 started proxy at port: 19398 + cd /tmp/tidb_cdc_test/http_proxies ++ run_cdc_cli_tso_query 10.233.93.110 2379 + pd_host=10.233.93.110 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://10.233.93.110:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.12838.out cli tso query --pd=http://10.233.93.110:2379 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:10 CST 2025] <<<<<< run test case autorandom success! >>>>>> [2025/04/04 15:49:10.861 +08:00] [WARN] [diff.go:557] ["checksum is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [sourceChecksum=0] [targetChecksum=3686522712] [getSourceChecksumCost=2.380936ms] [getTargetChecksumCost=7.570752ms] table test.`simple-dash` exists + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114370764439564, "checkpoint_time": "2025-04-04 15:49:00.233", "error": null }' + echo '{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114370764439564, "checkpoint_time": "2025-04-04 15:49:00.233", "error": null }' { "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114370764439564, "checkpoint_time": "2025-04-04 15:49:00.233", "error": null } ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114370764439564, '"checkpoint_time":' '"2025-04-04' '15:49:00.233",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114370764439564, '"checkpoint_time":' '"2025-04-04' '15:49:00.233",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: https://127.0.0.1:2579 + set +x failed proxying s2c rpc error: code = Canceled desc = context canceled + tso='457114373372248065 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114373372248065 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114373372248065 + echo 'query start ts: 457114373372248065' query start ts: 457114373372248065 + run_cdc_server --workdir /tmp/tidb_cdc_test/http_proxies --binary cdc.test [Fri Apr 4 15:49:11 CST 2025] <<<<<< START cdc server in http_proxies case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.1286912871.out server --log-file /tmp/tidb_cdc_test/http_proxies/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_proxies/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Uses proxy env variable http_proxy == 'http://127.0.0.1:19398' * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 19398 (#0) * Server auth using Basic with user 'ticdc' > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 502 Bad Gateway < Date: Fri, 04 Apr 2025 07:49:11 GMT < Content-Length: 0 < * Connection #0 to host 127.0.0.1 left intact + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42170.out cli changefeed --changefeed-id custom-changefeed-name pause [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } PASS [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Uses proxy env variable http_proxy == 'http://127.0.0.1:19398' * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 19398 (#0) * Server auth using Basic with user 'ticdc' > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 OK < Content-Length: 814 < Content-Type: text/plain; charset=utf-8 < Date: Fri, 04 Apr 2025 07:49:14 GMT < { [814 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 {"id":"573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752951} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/5a4295ffc5dd307d 573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 /tidb/cdc/default/default/upstream/7489361733787761564 {"id":7489361733787761564,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.93.110:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 {"id":"573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752951} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/5a4295ffc5dd307d 573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 /tidb/cdc/default/default/upstream/7489361733787761564 {"id":7489361733787761564,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.93.110:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 {"id":"573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752951} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/5a4295ffc5dd307d 573a1cb9-a96d-44a2-a4f1-9aa82d72f1e0 /tidb/cdc/default/default/upstream/7489361733787761564 {"id":7489361733787761564,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.93.110:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + echo started cdc server successfully started cdc server successfully + SINK_URI=blackhole:/// + run_cdc_cli changefeed create --start-ts=457114373372248065 --sink-uri=blackhole:/// + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.12933.out cli changefeed create --start-ts=457114373372248065 --sink-uri=blackhole:/// Create changefeed successfully! ID: 8ec685c9-a519-4236-b47c-2a3c08f7606e Info: {"upstream_id":7489361733787761564,"namespace":"default","id":"8ec685c9-a519-4236-b47c-2a3c08f7606e","sink_uri":"blackhole:///","create_time":"2025-04-04T15:49:15.080623065+08:00","start_ts":457114373372248065,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114373372248065,"checkpoint_ts":457114373372248065,"checkpoint_time":"2025-04-04 15:49:10.181"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:49:14.892 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] [2025/04/04 15:49:15.114 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:49:15.119 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:49:15.306 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:49:15.308 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:49:15.308 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:49:15.309 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:49:15.502 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:49:15.506 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:49:15.700 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:49:15.702 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:49:15.703 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:49:15.705 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:49:15.898 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:49:15.912 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:49:16.094 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:49:16.096 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:49:16.112 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:49:16.113 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:49:16.288 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:49:16.301 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:49:16.481 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:49:16.483 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:49:16.484 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:49:16.485 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] + set +x + sleep 5 [2025/04/04 15:49:16.823 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:49:16.870 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:49:17.048 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:49:17.049 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:49:17.074 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:49:17.076 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:49:17.284 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:49:17.390 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114373910167558, "checkpoint_time": "2025-04-04 15:49:12.233", "error": null }' + echo '{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114373910167558, "checkpoint_time": "2025-04-04 15:49:12.233", "error": null }' { "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114373910167558, "checkpoint_time": "2025-04-04 15:49:12.233", "error": null } ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114373910167558, '"checkpoint_time":' '"2025-04-04' '15:49:12.233",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114373910167558, '"checkpoint_time":' '"2025-04-04' '15:49:12.233",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42244.out cli changefeed update --pd=https://127.0.0.1:2579 --config=/tmp/tidb_cdc_test/cli_tls_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name [2025/04/04 15:49:17.634 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:49:17.635 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:49:17.670 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:49:17.673 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:49:17.847 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:49:17.885 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:49:18.216 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:49:18.217 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:49:18.285 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:49:18.287 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361821928155017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:05.864262803+08:00","start_ts":457114370358378497,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114373910167558,"checkpoint_time":"2025-04-04 15:49:12.233"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... check_redo_resolved_ts a3a0ae83-847b-4d93-8b79-bd65d93fe849 457114375125467138 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/a3a0ae83-847b-4d93-8b79-bd65d93fe849/meta global resolved ts 457114374692929540 not forward to 457114375125467138 run task failed 1-th time, retry later [2025/04/04 15:49:18.451 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:49:18.503 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:49:18.662 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:49:18.664 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:49:18.796 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:49:18.799 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] check_redo_resolved_ts a3a0ae83-847b-4d93-8b79-bd65d93fe849 457114375125467138 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/a3a0ae83-847b-4d93-8b79-bd65d93fe849/meta [2025/04/04 15:49:18.990 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:49:19.038 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] run task successfully [2025/04/04 15:49:19.190 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:49:19.191 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:49:19.246 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:49:19.248 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:49:19.476 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:49:19.602 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] + set +x wait process cdc.test exit for 1-th time... [2025/04/04 15:49:19.732 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:49:19.733 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:49:19.806 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:49:19.808 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42286.out cli changefeed --changefeed-id custom-changefeed-name resume wait process cdc.test exit for 2-th time... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/changefeed_dup_error_restart Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2025/04/04 15:49:20.744 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:49:20.745 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:49:20.745 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:49:20.745 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:49:20.745 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114369830158340] [resolvedTs=457114375216955398] [2025/04/04 15:49:20.746 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:49:20.746 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:49:20.746 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:49:20.747 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.30085ms] [2025/04/04 15:49:20.747 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.37448ms] [2025/04/04 15:49:20.752 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:49:20.755 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:49:20.756 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:49:20.757 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:49:20.757 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:49:20.757 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:49:20.757 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:49:20.761 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:49:20.763 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:49:20.763 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:49:20.764 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114369817051168,\"CommitTs\":457114369830158340,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file_large_value`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file_large_value`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_file_large_value\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:49:20.964 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114375216955398] [2025/04/04 15:49:20.964 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114375216955398] [2025/04/04 15:49:21.265 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114375216955398] [2025/04/04 15:49:21.265 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114375216955398] [2025/04/04 15:49:21.366 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=8] [appliedDDLCount=0] [currentCheckpoint=457114375216955398] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:49:21.366 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] Apply redo log successfully + set +x + check + services=($(cat $WORK_DIR/test_proxy.log | xargs -L1 dirname | sort | uniq)) ++ cat /tmp/tidb_cdc_test/http_proxies/test_proxy.log ++ xargs -L1 dirname ++ sort ++ uniq + service_type_count=29 + echo 'captured services: ' captured services: + echo . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv '[2025/04' '[metadata=MD{:authority=[10.233.93.110:2379],' 'user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'content-type=[application' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.93.110:2379],' 'grpc-accept-encoding=[gzip],' 'user-agent=[grpc-go' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.93.110:2379],' 'user-agent=[grpc-go' '[metadata=MD{grpc-accept-encoding=[gzip],' 'features=[stream-multiplexing],' 'content-type=[application/grpc],' ':authority=[10.233.93.110:20162],' 'user-agent=[grpc-go' '[metadata=MD{user-agent=[grpc-go/1.64.0],' 'content-type=[application' '[method=/cdcpb.ChangeData' '[method=/pdpb.PD' '[method=/resource_manager.ResourceManager' '[stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0' . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv [2025/04 [metadata=MD{:authority=[10.233.93.110:2379], user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], content-type=[application [metadata=MD{content-type=[application/grpc], :authority=[10.233.93.110:2379], grpc-accept-encoding=[gzip], user-agent=[grpc-go [metadata=MD{content-type=[application/grpc], :authority=[10.233.93.110:2379], user-agent=[grpc-go [metadata=MD{grpc-accept-encoding=[gzip], features=[stream-multiplexing], content-type=[application/grpc], :authority=[10.233.93.110:20162], user-agent=[grpc-go [metadata=MD{user-agent=[grpc-go/1.64.0], content-type=[application [method=/cdcpb.ChangeData [method=/pdpb.PD [method=/resource_manager.ResourceManager [stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0 + '[' 29 -ge 2 ']' + check_logs /tmp/tidb_cdc_test/http_proxies ++ date + echo '[Fri Apr 4 15:49:21 CST 2025] <<<<<< run test case http_proxies success! >>>>>>' [Fri Apr 4 15:49:21 CST 2025] <<<<<< run test case http_proxies success! >>>>>> + stop_tidb_cluster failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled check diff successfully [Fri Apr 4 15:49:21 CST 2025] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> count(*) 0 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114377069264898, "checkpoint_time": "2025-04-04 15:49:24.284", "error": null }' + echo '{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114377069264898, "checkpoint_time": "2025-04-04 15:49:24.284", "error": null }' { "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114377069264898, "checkpoint_time": "2025-04-04 15:49:24.284", "error": null } ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114377069264898, '"checkpoint_time":' '"2025-04-04' '15:49:24.284",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114377069264898, '"checkpoint_time":' '"2025-04-04' '15:49:24.284",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42358.out cli changefeed --changefeed-id custom-changefeed-name remove Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114377069264898 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:49:25.821 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] [2025/04/04 15:49:26.023 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:49:26.029 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:49:26.372 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:49:26.374 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:49:26.375 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:49:26.376 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:49:26.547 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:49:26.551 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:49:26.866 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:49:26.868 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:49:26.878 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:49:26.879 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:49:27.079 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:49:27.112 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] + set +x [2025/04/04 15:49:27.443 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:49:27.446 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:49:27.491 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:49:27.492 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:49:27.711 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:49:27.788 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:49:28.094 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:49:28.098 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:49:28.261 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:49:28.264 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:49:28.603 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:49:28.896 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:49:28.992 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:49:28.995 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:49:29.340 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:49:29.367 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:49:29.369 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:49:29.768 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:49:29.772 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:49:29.887 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361947913481648 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1ad7e40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:17384, start at 2025-04-04 15:49:27.674967327 +0800 CST m=+2.233439850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:27.683 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:27.673 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:27.673 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361947913481648 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1ad7e40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:17384, start at 2025-04-04 15:49:27.674967327 +0800 CST m=+2.233439850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:27.683 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:27.673 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:27.673 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361942220038574 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1ad724000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:17456, start at 2025-04-04 15:49:27.666050064 +0800 CST m=+2.124589393 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:27.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:27.671 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:27.671 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore [2025/04/04 15:49:30.138 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:49:30.343 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:49:30.345 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:49:30.507 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:49:30.512 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] changefeed count 0 check pass, pd_addr: https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42427.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [2025/04/04 15:49:30.771 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361821928155017,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:30.965833942+08:00","start_ts":457114378786308100,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114378786308100,"checkpoint_ts":457114378786308100,"checkpoint_time":"2025-04-04 15:49:30.834"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.18794.out cli tso query --pd=http://127.0.0.1:2379 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend + set +x Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361967815037071 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1b1ec80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:32488, start at 2025-04-04 15:49:32.222343719 +0800 CST m=+2.043470084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:32.230 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:32.210 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:32.210 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361967815037071 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1b1ec80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:32488, start at 2025-04-04 15:49:32.222343719 +0800 CST m=+2.043470084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:32.230 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:32.210 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:32.210 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330E1738841816 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:49:33 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact + set +x + tso='457114379137581057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114379137581057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "mysql.user"="root" "mysql.port"="4000" "workload"="core" "insertproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" "updateproportion"="0" "mysql.db"="changefeed_dup_error_restart" "dotransactions"="false" "recordcount"="10" "threadcount"="2" "operationcount"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 8.96781ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1755.3, Avg(us): 1697, Min(us): 975, Max(us): 3241, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:49:33 CST 2025] <<<<<< START cdc server in changefeed_dup_error_restart case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLDuplicateEntryError=5%return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.1884918851.out server --log-file /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... + stop_proxy + kill 9089 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361959504766523 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1b21f00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-1hc17-b4qpf-kslkl, pid:32566, start at 2025-04-04 15:49:32.415337015 +0800 CST m=+2.163347701 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:32.422 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:32.412 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:32.412 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114379690442755, "checkpoint_time": "2025-04-04 15:49:34.283", "error": null }' + echo '{ "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114379690442755, "checkpoint_time": "2025-04-04 15:49:34.283", "error": null }' { "upstream_id": 7489361821928155017, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114379690442755, "checkpoint_time": "2025-04-04 15:49:34.283", "error": null } ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114379690442755, '"checkpoint_time":' '"2025-04-04' '15:49:34.283",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361821928155017, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114379690442755, '"checkpoint_time":' '"2025-04-04' '15:49:34.283",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42533.out cli unsafe delete-service-gc-safepoint Confirm that you know what this command will do and use it at your own risk [Y/N] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.33949.out cli tso query --pd=http://127.0.0.1:2379 CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:36 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dbe9b109-390c-4569-bb63-fa5e82c364c5 {"id":"dbe9b109-390c-4569-bb63-fa5e82c364c5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752974} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc69b32f2 dbe9b109-390c-4569-bb63-fa5e82c364c5 /tidb/cdc/default/default/upstream/7489361947913481648 {"id":7489361947913481648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dbe9b109-390c-4569-bb63-fa5e82c364c5 {"id":"dbe9b109-390c-4569-bb63-fa5e82c364c5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752974} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc69b32f2 dbe9b109-390c-4569-bb63-fa5e82c364c5 /tidb/cdc/default/default/upstream/7489361947913481648 {"id":7489361947913481648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dbe9b109-390c-4569-bb63-fa5e82c364c5 {"id":"dbe9b109-390c-4569-bb63-fa5e82c364c5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752974} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc69b32f2 dbe9b109-390c-4569-bb63-fa5e82c364c5 /tidb/cdc/default/default/upstream/7489361947913481648 {"id":7489361947913481648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.18910.out cli changefeed create --start-ts=457114379137581057 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' start tidb cluster in /tmp/tidb_cdc_test/sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Create changefeed successfully! ID: 4a3995c2-fa26-4300-88a3-c622b4c5c41a Info: {"upstream_id":7489361947913481648,"namespace":"default","id":"4a3995c2-fa26-4300-88a3-c622b4c5c41a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:49:37.163595374+08:00","start_ts":457114379137581057,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114379137581057,"checkpoint_ts":457114379137581057,"checkpoint_time":"2025-04-04 15:49:32.174"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_s3 Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42566.out cli unsafe reset --no-confirm --pd=https://127.0.0.1:2579 + set +x + tso='457114380379357185 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114380379357185 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:49:38 CST 2025] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.3397933981.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:49:38.295 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m6.617603645s"] Verifying downstream PD is started... table mark.finish_mark_0 exists table mark.finish_mark_1 exists table mark.finish_mark_2 exists table mark.finish_mark_3 exists table mark.finish_mark_4 exists table mark.finish_mark_5 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table mark.finish_mark_5 exists table mark.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 {"id":"cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752978} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6a7d7e3 cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 /tidb/cdc/default/default/upstream/7489361967815037071 {"id":7489361967815037071,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 {"id":"cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752978} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6a7d7e3 cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 /tidb/cdc/default/default/upstream/7489361967815037071 {"id":7489361967815037071,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 {"id":"cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752978} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6a7d7e3 cb9b8d4f-122d-42e9-a8b2-65e6a2cb59d9 /tidb/cdc/default/default/upstream/7489361967815037071 {"id":7489361967815037071,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.34038.out cli changefeed create --start-ts=457114380379357185 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: dea8efba-0ae8-4c4e-a9ae-caaa7f87a6e0 Info: {"upstream_id":7489361967815037071,"namespace":"default","id":"dea8efba-0ae8-4c4e-a9ae-caaa7f87a6e0","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:41.871191829+08:00","start_ts":457114380379357185,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114380379357185,"checkpoint_ts":457114380379357185,"checkpoint_time":"2025-04-04 15:49:36.911"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42668.out cli unsafe resolve-lock --region=10 + set +x table generate_column.t1 not exists for 1-th check, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.42690.out cli unsafe resolve-lock --region=10 --ts=457114381210877954 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table generate_column.t1 exists table generate_column.t2 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362015491739633 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1bd5f0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:13954, start at 2025-04-04 15:49:43.944595169 +0800 CST m=+2.081266660 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:43.953 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:43.932 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:43.932 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362015491739633 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1bd5f0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:13954, start at 2025-04-04 15:49:43.944595169 +0800 CST m=+2.081266660 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:43.953 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:43.932 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:43.932 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362016736007468 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1bd8c8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-87zdl-ttvzb-nqm40, pid:14019, start at 2025-04-04 15:49:44.140738229 +0800 CST m=+2.177048016 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:44.148 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:44.114 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:44.114 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:49:46.199 +08:00] [DEBUG] [peer.rs:2639] ["renew lease"] [reason=tick] [peer_id=51] [region_id=50] [thread_id=92] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sequence/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 11777 333 --:--:-- --:--:-- --:--:-- 12111 { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362016664933514 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1be6c8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:15800, start at 2025-04-04 15:49:45.026428912 +0800 CST m=+2.157985745 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:45.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:45.010 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:45.010 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362016664933514 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1be6c8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:15800, start at 2025-04-04 15:49:45.026428912 +0800 CST m=+2.157985745 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:45.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:45.010 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:45.010 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362019161333582 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1be6bc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:15866, start at 2025-04-04 15:49:45.049390749 +0800 CST m=+2.105683429 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:45.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:45.107 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:45.107 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table generate_column.t2 not exists for 2-th check, retry later [Fri Apr 4 15:49:48 CST 2025] <<<<<< START cdc server in sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.1542615428.out server --log-file /tmp/tidb_cdc_test/sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sequence/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:49:49 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1712717129.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table generate_column.t2 exists check diff successfully } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 10166 0 --:--:-- --:--:-- --:--:-- 10166 { "version": "v9.0.0-beta.1-10-ge8bfbe4ba", "git_hash": "e8bfbe4ba98e05140ab733ee3d08dd3b99918721", "id": "a98663fd-6a54-45aa-916e-1217af9d7bfe", "pid": 41919, "is_owner": true wait process cdc.test exit for 1-th time... }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:51 CST 2025] <<<<<< run test case generate_column success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:51 CST 2025] <<<<<< run test case cli_tls_with_auth success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:51 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d5f81cc-f862-498e-a1b8-5d451ae84b0e {"id":"5d5f81cc-f862-498e-a1b8-5d451ae84b0e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752988} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6db3dda 5d5f81cc-f862-498e-a1b8-5d451ae84b0e /tidb/cdc/default/default/upstream/7489362015491739633 {"id":7489362015491739633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d5f81cc-f862-498e-a1b8-5d451ae84b0e {"id":"5d5f81cc-f862-498e-a1b8-5d451ae84b0e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752988} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6db3dda 5d5f81cc-f862-498e-a1b8-5d451ae84b0e /tidb/cdc/default/default/upstream/7489362015491739633 {"id":7489362015491739633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d5f81cc-f862-498e-a1b8-5d451ae84b0e {"id":"5d5f81cc-f862-498e-a1b8-5d451ae84b0e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752988} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6db3dda 5d5f81cc-f862-498e-a1b8-5d451ae84b0e /tidb/cdc/default/default/upstream/7489362015491739633 {"id":7489362015491739633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.cli.15481.out cli changefeed create --start-ts=457114383369895937 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sequence/conf/force_replicate.toml Create changefeed successfully! ID: 83e0e5cc-deca-48af-8569-2fd56f068334 Info: {"upstream_id":7489362015491739633,"namespace":"default","id":"83e0e5cc-deca-48af-8569-2fd56f068334","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:49:51.899886874+08:00","start_ts":457114383369895937,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114383369895937,"checkpoint_ts":457114383369895937,"checkpoint_time":"2025-04-04 15:49:48.319"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:52 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 {"id":"5fc2a483-b48f-4f81-b50f-b57d0b58d7a5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752989} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd29d8 5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 {"id":"5fc2a483-b48f-4f81-b50f-b57d0b58d7a5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752989} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd29d8 5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 {"id":"5fc2a483-b48f-4f81-b50f-b57d0b58d7a5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752989} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd29d8 5fc2a483-b48f-4f81-b50f-b57d0b58d7a5 /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x The 1 times to try to start tidb cluster... ***************** properties ***************** "workload"="core" "recordcount"="5000" "fieldcount"="100" "operationcount"="0" "mysql.port"="4000" "dotransactions"="false" "updateproportion"="0" "mysql.host"="127.0.0.1" "insertproportion"="0" "mysql.user"="root" "readproportion"="0" "threadcount"="10" "scanproportion"="0" "mysql.db"="consistent_replicate_storage_s3" "readallfields"="true" "requestdistribution"="uniform" ********************************************** + set +x Run finished, takes 1.791275202s INSERT - Takes(s): 1.8, Count: 5000, OPS: 2802.0, Avg(us): 3246, Min(us): 1118, Max(us): 65515, 95th(us): 3000, 99th(us): 48000 table consistent_replicate_storage_s3.usertable not exists for 1-th check, retry later table sequence_test.t1 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table consistent_replicate_storage_s3.usertable exists table consistent_replicate_storage_s3.check1 not exists for 1-th check, retry later table sequence_test.t1 exists check table exists success check diff failed 1-th time, retry later table consistent_replicate_storage_s3.check1 not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully ERROR 1146 (42S02) at line 1: Table 'sequence_test.seq0' doesn't exist table sequence_test.mark_table exists ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:49:47.842","puller_resolved_ts":"2025-04-04 15:49:39.843","last_synced_ts":"2025-04-04 15:47:31.242","now_ts":"2025-04-04 15:49:48.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:49:47.842","puller_resolved_ts":"2025-04-04' '15:49:39.843","last_synced_ts":"2025-04-04' '15:47:31.242","now_ts":"2025-04-04' '15:49:48.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 14819 7.8 0.0 14968260 145652 ? Sl 15:47 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 14878 5.9 0.0 14967748 140232 ? Sl 15:47 0:09 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1743752994.8414695,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046781c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752994.841522,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752994.9649062,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0030d9500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752994.9649746,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752994.9723077,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005250540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752994.9723678,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} wait process cdc.test exit for 1-th time... table consistent_replicate_storage_s3.check1 not exists for 3-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) wait process cdc.test exit for 2-th time... [Pipeline] // cache Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:01 CST 2025] <<<<<< run test case sequence success! >>>>>> [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } {"level":"warn","ts":"2025-04-04T15:50:02.749529+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} table consistent_replicate_storage_s3.check1 not exists for 4-th check, retry later {"level":"warn","ts":"2025-04-04T15:50:02.754013+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:02.846578+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/debezium_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:02 CST 2025] <<<<<< run test case debezium_basic success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362102157714324 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d08d80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:26263, start at 2025-04-04 15:50:03.577703507 +0800 CST m=+2.331400312 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:03.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:03.574 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:03.574 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362102157714324 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d08d80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:26263, start at 2025-04-04 15:50:03.577703507 +0800 CST m=+2.331400312 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:03.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:03.574 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:03.574 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362102735316683 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d099c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:26336, start at 2025-04-04 15:50:03.663231802 +0800 CST m=+2.344868590 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:03.671 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:03.673 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:03.673 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:06 CST 2025] <<<<<< run test case kafka_simple_basic success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 6-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.27707.out cli tso query --pd=http://127.0.0.1:2379 {"level":"warn","ts":1743753007.9211783,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046781c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743753008.060193,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005250540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:08.750718+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:08.75515+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:08.847286+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table consistent_replicate_storage_s3.check1 not exists for 7-th check, retry later + set +x + tso='457114388522598401 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114388522598401 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:50:09 CST 2025] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.2773827740.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_dup_error_restart.finish_mark_1 exists check diff successfully ***************** properties ***************** "scanproportion"="0" "readproportion"="0" "mysql.user"="root" "mysql.port"="4000" "requestdistribution"="uniform" "readallfields"="true" "dotransactions"="false" "updateproportion"="0" "threadcount"="2" "workload"="core" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_dup_error_restart" "operationcount"="0" "recordcount"="10" ********************************************** Run finished, takes 3.926295ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3704.4, Avg(us): 688, Min(us): 427, Max(us): 1701, 95th(us): 2000, 99th(us): 2000 table consistent_replicate_storage_s3.check1 not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:10 CST 2025] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:50:12 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc5e482c-fc3d-466a-985b-35cde02f61d2 {"id":"dc5e482c-fc3d-466a-985b-35cde02f61d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753009} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc72517e2 dc5e482c-fc3d-466a-985b-35cde02f61d2 /tidb/cdc/default/default/upstream/7489362102157714324 {"id":7489362102157714324,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc5e482c-fc3d-466a-985b-35cde02f61d2 {"id":"dc5e482c-fc3d-466a-985b-35cde02f61d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753009} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc72517e2 dc5e482c-fc3d-466a-985b-35cde02f61d2 /tidb/cdc/default/default/upstream/7489362102157714324 {"id":7489362102157714324,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc5e482c-fc3d-466a-985b-35cde02f61d2 {"id":"dc5e482c-fc3d-466a-985b-35cde02f61d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753009} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc72517e2 dc5e482c-fc3d-466a-985b-35cde02f61d2 /tidb/cdc/default/default/upstream/7489362102157714324 {"id":7489362102157714324,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.27793.out cli changefeed create --start-ts=457114388522598401 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 Create changefeed successfully! ID: d0f8560d-764d-48ae-a4fc-52f295ad3f77 Info: {"upstream_id":7489362102157714324,"namespace":"default","id":"d0f8560d-764d-48ae-a4fc-52f295ad3f77","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:50:12.937578313+08:00","start_ts":457114388522598401,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114388522598401,"checkpoint_ts":457114388522598401,"checkpoint_time":"2025-04-04 15:50:07.975"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table consistent_replicate_storage_s3.check1 not exists for 9-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 + set +x 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:14.751355+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:14.756118+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:14.848198+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table consistent_replicate_storage_s3.check1 not exists for 10-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:15 CST 2025] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 11-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 table consistent_replicate_storage_s3.check1 not exists for 12-th check, retry later [2025/04/04 15:50:19.658 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 21264 0 --:--:-- --:--:-- --:--:-- 21264 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:47:57.983","puller_resolved_ts":"2025-04-04 15:47:57.983","last_synced_ts":"2025-04-04 15:47:51.033","now_ts":"2025-04-04 15:50:08.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:57.983","puller_resolved_ts":"2025-04-04' '15:47:57.983","last_synced_ts":"2025-04-04' '15:47:51.033","now_ts":"2025-04-04' '15:50:08.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:47:57.983","puller_resolved_ts":"2025-04-04' '15:47:57.983","last_synced_ts":"2025-04-04' '15:47:51.033","now_ts":"2025-04-04' '15:50:08.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [2025/04/04 15:50:19.970 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:50:19.976 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:19 CST 2025] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> [2025/04/04 15:50:20.337 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:50:20.344 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:50:20.352 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:50:20.356 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 [2025/04/04 15:50:20.713 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:50:20.721 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:20.752501+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:20.757424+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:20.84968+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} [2025/04/04 15:50:21.000 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:50:21.007 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:50:21.083 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:50:21.099 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:50:21.368 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:50:21.369 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] table consistent_replicate_storage_s3.check1 not exists for 13-th check, retry later [2025/04/04 15:50:21.744 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:50:21.746 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:50:21.752 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:50:21.755 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:50:22.129 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:50:22.130 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:50:22.507 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:50:22.511 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:50:22.524 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:50:22.565 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:22.739677+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:50:22.73972+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:50:22.744345+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:50:22.744374+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:50:22.809197+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:50:22.80923+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:50:22.944 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:50:22.945 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:50:23.323 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:50:23.325 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:50:23.338 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:50:23.353 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] table consistent_replicate_storage_s3.check1 not exists for 14-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:50:23.625 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:50:23.634 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:50:23.991 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:50:24.006 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:50:24.016 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:50:24.044 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:50:24.267 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:50:24.380 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:24 CST 2025] <<<<<< run test case kafka_simple_claim_check success! >>>>>> table consistent_replicate_storage_s3.check1 exists table consistent_replicate_storage_s3.t2 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:26.754384+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:26.758549+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:26.850982+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table consistent_replicate_storage_s3.t2 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:28 CST 2025] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> table consistent_replicate_storage_s3.t2 not exists for 3-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 [2025/04/04 15:50:29.750 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362204383354546 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1e81140009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:14696, start at 2025-04-04 15:50:27.666808887 +0800 CST m=+2.221693514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:27.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:27.677 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:27.677 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362204383354546 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1e81140009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:14696, start at 2025-04-04 15:50:27.666808887 +0800 CST m=+2.221693514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:27.674 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:27.677 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:27.677 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362203444551751 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1e82ec000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:14752, start at 2025-04-04 15:50:27.792850201 +0800 CST m=+2.277667773 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:27.801 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:27.771 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:27.771 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1743753029.8425221,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0046781c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753029.842565,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:50:29.978 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:50:29.998 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] {"level":"warn","ts":1743753029.9660387,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0030d9500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753029.966075,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743753029.973429,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005250540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743753029.9734654,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:50:30.184 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:50:30.203 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:50:30.236 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:50:30.246 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:50:30.391 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:50:30.471 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:50:30.610 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:50:30.627 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:50:30.709 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:50:30.727 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:50:30.826 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:50:30.930 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:50:31.018 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:50:31.041 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:50:31.148 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:50:31.173 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:50:31.232 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:50:31.372 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:50:31.437 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:50:31.473 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] table consistent_replicate_storage_s3.t2 not exists for 4-th check, retry later [2025/04/04 15:50:31.595 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:50:31.619 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:50:31.650 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:50:31.815 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:50:31.868 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:50:31.911 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:50:32.032 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:50:32.055 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:50:32.085 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:50:32.253 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:50:32.310 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.15959.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:50:32.347 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:50:32.482 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:50:32.517 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:50:32.530 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:50:32.754 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:50:32.801 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:50:32.835 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:32 CST 2025] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:50:32.755188+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:32.760036+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:32.851956+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} [2025/04/04 15:50:32.970 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:50:33.008 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:50:33.016 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:50:33.170 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:50:33.238 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:50:33.278 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] table consistent_replicate_storage_s3.t2 not exists for 5-th check, retry later [2025/04/04 15:50:33.380 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:50:33.417 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:50:33.452 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:50:33.579 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] + set +x + tso='457114394853900289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114394853900289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114394853900289 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:50:33 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1598915991.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:50:33.663 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:50:33.711 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:50:33.803 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:50:33.842 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:50:33.870 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:50:33.996 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:50:34.060 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:50:34.102 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:50:34.187 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:50:34.236 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:50:34.241 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:50:34.387 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:50:34.428 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:50:34.469 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:50:34.622 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:50:34.638 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:50:34.685 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] table consistent_replicate_storage_s3.t2 not exists for 6-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:50:36 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd0af470-3e15-4fbd-8a62-2f4a42d86d56 {"id":"fd0af470-3e15-4fbd-8a62-2f4a42d86d56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753033} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc783a1e5 fd0af470-3e15-4fbd-8a62-2f4a42d86d56 /tidb/cdc/default/default/upstream/7489362204383354546 {"id":7489362204383354546,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd0af470-3e15-4fbd-8a62-2f4a42d86d56 {"id":"fd0af470-3e15-4fbd-8a62-2f4a42d86d56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753033} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc783a1e5 fd0af470-3e15-4fbd-8a62-2f4a42d86d56 /tidb/cdc/default/default/upstream/7489362204383354546 {"id":7489362204383354546,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd0af470-3e15-4fbd-8a62-2f4a42d86d56 {"id":"fd0af470-3e15-4fbd-8a62-2f4a42d86d56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753033} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc783a1e5 fd0af470-3e15-4fbd-8a62-2f4a42d86d56 /tidb/cdc/default/default/upstream/7489362204383354546 {"id":7489362204383354546,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114394853900289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16027.out cli changefeed create --start-ts=457114394853900289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362204383354546,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:50:37.024404217+08:00","start_ts":457114394853900289,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114394853900289,"checkpoint_ts":457114394853900289,"checkpoint_time":"2025-04-04 15:50:32.127"} PASS table consistent_replicate_storage_s3.t2 exists coverage: 2.5% of statements in github.com/pingcap/tiflow/... 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:37 CST 2025] <<<<<< run test case canal_json_basic success! >>>>>> check diff successfully wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later {"level":"warn","ts":"2025-04-04T15:50:38.756491+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:50:38.760402+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080d6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} wait process cdc.test exit for 2-th time... {"level":"warn","ts":"2025-04-04T15:50:38.85339+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c97880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:39 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1745517457.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... [2025/04/04 15:50:39.903 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2025/04/04 15:50:40.116 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:50:40.134 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster [2025/04/04 15:50:40.308 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:50:40.310 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:50:40.344 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:50:40.346 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] table test.t1 exists + sleep 5 [2025/04/04 15:50:40.507 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:50:40.576 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:50:40.709 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:50:40.711 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:50:40.789 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:50:40.790 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:50:40.907 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:50:41.007 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:50:41.109 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:50:41.111 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:50:41.215 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:50:41.217 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:50:41.301 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:50:41.427 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:50:41.498 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:50:41.500 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:50:41.641 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:50:41.643 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:50:41.688 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:50:41.855 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:50:41.886 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:50:41.888 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:50:42.070 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:50:42.073 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:50:42.078 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:41 CST 2025] <<<<<< run test case canal_json_content_compatible success! >>>>>> [2025/04/04 15:50:42.274 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:50:42.276 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:50:42.288 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] table changefeed_dup_error_restart.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:40 CST 2025] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> [2025/04/04 15:50:42.463 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:50:42.492 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:50:42.494 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:50:42.672 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:50:42.675 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:50:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8190197b-49ea-4b91-a651-d9b42f44d140 {UpstreamID:7489362016664933514 Namespace:default ID:8190197b-49ea-4b91-a651-d9b42f44d140 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:49:52.508872121 +0800 CST StartTs:457114384442327046 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031efc20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114384455434248} {CheckpointTs:457114385005936659 MinTableBarrierTs:457114397470883844 AdminJobType:noop} span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1603d7f-0108-495f-98dd-0d19d7de1ea7 {"id":"d1603d7f-0108-495f-98dd-0d19d7de1ea7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753039} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd2bc8 d1603d7f-0108-495f-98dd-0d19d7de1ea7 /tidb/cdc/default/default/changefeed/info/8190197b-49ea-4b91-a651-d9b42f44d140 {"upstream-id":7489362016664933514,"namespace":"default","changefeed-id":"8190197b-49ea-4b91-a651-d9b42f44d140","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:49:52.508872121+08:00","start-ts":457114384442327046,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114384455434248} /tidb/cdc/default/default/changefeed/status/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":457114385005936659,"min-table-barrier-ts":457114397470883844,"admin-job-type":0} /tidb/cdc/default/default/task/position/d1603d7f-0108-495f-98dd-0d19d7de1ea7/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8190197b-49ea-4b91-a651-d9b42f44d140 {UpstreamID:7489362016664933514 Namespace:default ID:8190197b-49ea-4b91-a651-d9b42f44d140 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:49:52.508872121 +0800 CST StartTs:457114384442327046 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031efc20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114384455434248} {CheckpointTs:457114385005936659 MinTableBarrierTs:457114397470883844 AdminJobType:noop} span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1603d7f-0108-495f-98dd-0d19d7de1ea7 {"id":"d1603d7f-0108-495f-98dd-0d19d7de1ea7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753039} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd2bc8 d1603d7f-0108-495f-98dd-0d19d7de1ea7 /tidb/cdc/default/default/changefeed/info/8190197b-49ea-4b91-a651-d9b42f44d140 {"upstream-id":7489362016664933514,"namespace":"default","changefeed-id":"8190197b-49ea-4b91-a651-d9b42f44d140","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:49:52.508872121+08:00","start-ts":457114384442327046,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114384455434248} /tidb/cdc/default/default/changefeed/status/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":457114385005936659,"min-table-barrier-ts":457114397470883844,"admin-job-type":0} /tidb/cdc/default/default/task/position/d1603d7f-0108-495f-98dd-0d19d7de1ea7/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8190197b-49ea-4b91-a651-d9b42f44d140 {UpstreamID:7489362016664933514 Namespace:default ID:8190197b-49ea-4b91-a651-d9b42f44d140 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:49:52.508872121 +0800 CST StartTs:457114384442327046 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031efc20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114384455434248} {CheckpointTs:457114385005936659 MinTableBarrierTs:457114397470883844 AdminJobType:noop} span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114397208739844, checkpointTs: 457114385005936659, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114385005936659, checkpointTs: 457114385005936659, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1603d7f-0108-495f-98dd-0d19d7de1ea7 {"id":"d1603d7f-0108-495f-98dd-0d19d7de1ea7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753039} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6dd2bc8 d1603d7f-0108-495f-98dd-0d19d7de1ea7 /tidb/cdc/default/default/changefeed/info/8190197b-49ea-4b91-a651-d9b42f44d140 {"upstream-id":7489362016664933514,"namespace":"default","changefeed-id":"8190197b-49ea-4b91-a651-d9b42f44d140","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:49:52.508872121+08:00","start-ts":457114384442327046,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114384455434248} /tidb/cdc/default/default/changefeed/status/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":457114385005936659,"min-table-barrier-ts":457114397470883844,"admin-job-type":0} /tidb/cdc/default/default/task/position/d1603d7f-0108-495f-98dd-0d19d7de1ea7/8190197b-49ea-4b91-a651-d9b42f44d140 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362016664933514 {"id":7489362016664933514,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:50:42.713 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:50:42.874 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:50:42.921 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:50:42.923 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:50:43.076 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:50:43.079 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:50:43.131 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:50:43.270 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:50:43.340 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:50:43.342 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:50:43.460 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:50:43.465 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:50:43.548 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:50:43.659 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:50:43.753 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:50:43.755 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:50:43.857 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:50:43.858 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:50:43.964 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:50:44.065 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:50:44.185 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:50:44.186 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:50:44.263 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:50:44.265 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:50:44.388 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:50:44.455 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:50:44.616 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:50:44.618 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:50:44.683 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:50:44.685 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:50:44.831 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:50:44.882 +08:00] [INFO] [main.go:221] ["1 insert success: 2500"] + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 14696 9.6 0.0 2445844 216720 ? Sl 15:50 0:01 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753025434217895.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 14705 3.7 0.0 2888332 191260 ? Sl 15:50 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753025441997554.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 14752 10.3 0.0 3036980 216664 ? Sl 15:50 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753025503108742.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 7593 0 --:--:-- --:--:-- --:--:-- 7593 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:50:42.476","puller_resolved_ts":"2025-04-04 15:50:38.426","last_synced_ts":"2025-04-04 15:50:38.476","now_ts":"2025-04-04 15:50:45.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:50:42.476","puller_resolved_ts":"2025-04-04' '15:50:38.426","last_synced_ts":"2025-04-04' '15:50:38.476","now_ts":"2025-04-04' '15:50:45.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:50:42.476","puller_resolved_ts":"2025-04-04' '15:50:38.426","last_synced_ts":"2025-04-04' '15:50:38.476","now_ts":"2025-04-04' '15:50:45.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:45 CST 2025] <<<<<< run test case multi_topics success! >>>>>> [2025/04/04 15:50:41.609 +08:00] [WARN] [diff.go:766] ["rows is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [cost=1m30.748129112s] [2025/04/04 15:50:41.612 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] [2025/04/04 15:50:47.182 +08:00] [INFO] [dailytest.go:68] ["test pass!!!"] + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:49 CST 2025] <<<<<< run test case cdc success! >>>>>> [2025/04/04 15:50:50.238 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2025/04/04 15:50:50.464 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:50:50.476 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:50 CST 2025] <<<<<< run test case avro_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/safe_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:50:50.696 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:50:50.697 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:50:50.706 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:50:50.707 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:50:50.905 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:50:50.932 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:50:51.103 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:50:51.105 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:50:51.284 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:50:51.285 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:50:51.427 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:50:51.489 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:50:51.632 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:50:51.634 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:50:51.698 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:50:51.699 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:50:51.831 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:50:51.899 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:50:52.047 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:50:52.049 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:50:52.106 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:50:52.106 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:50:52.255 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:50:52.313 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:50:52.547 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:50:52.549 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:50:52.606 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:50:52.607 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:50:52.779 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:50:52.837 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:50:52.985 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:50:52.987 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:50:53.045 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:50:53.046 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] start tidb cluster in /tmp/tidb_cdc_test/safe_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:50:53.180 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:50:53.248 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:50:53.377 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:50:53.379 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:50:53.453 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:50:53.454 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:50:53.850 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:50:53.948 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:50:54.048 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:50:54.050 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:50:54.150 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:50:54.151 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:50:54.249 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:50:54.453 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:50:54.549 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:50:54.564 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:54 CST 2025] <<<<<< run test case canal_json_handle_key_only success! >>>>>> [2025/04/04 15:50:54.670 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:50:54.671 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:50:54.873 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:50:54.983 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:50:55.087 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:50:55.089 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:50:55.190 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:50:55.191 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:50:58 CST 2025] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362332927618167 TiDB Cluster ID. tikv_gc_leader_uuid 657ff205b30000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:18144, start at 2025-04-04 15:50:58.045567248 +0800 CST m=+1.973567381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:58.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:58.046 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:58.046 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362332927618167 TiDB Cluster ID. tikv_gc_leader_uuid 657ff205b30000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:18144, start at 2025-04-04 15:50:58.045567248 +0800 CST m=+1.973567381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:58.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:58.046 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:58.046 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362336780593841 TiDB Cluster ID. tikv_gc_leader_uuid 657ff205ef8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:18201, start at 2025-04-04 15:50:58.265269297 +0800 CST m=+2.114759251 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:58.272 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:58.238 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:58.238 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:51:00.457 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:51:00.830 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:51:00.839 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:51:01.060 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:51:01.065 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:51:01.083 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:51:01.084 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:51:01.254 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:51:01.283 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:51:01.453 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:51:01.455 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:51:01.486 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:51:01.487 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:51:01.645 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:51:01.694 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362342483942439 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2077c40017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:20054, start at 2025-04-04 15:50:59.859957367 +0800 CST m=+2.081571960 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:59.867 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:59.865 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:59.865 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362342483942439 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2077c40017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:20054, start at 2025-04-04 15:50:59.859957367 +0800 CST m=+2.081571960 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:59.867 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:59.865 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:59.865 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362335136818251 TiDB Cluster ID. tikv_gc_leader_uuid 657ff207b780006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-s8jj0-xfl8k-bjnbm, pid:20138, start at 2025-04-04 15:51:00.069682797 +0800 CST m=+2.210600585 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:00.077 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:00.062 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:00.062 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:51:01.843 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:51:01.845 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:51:01.902 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:51:01.903 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:51:02.042 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/safe_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/safe_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:51:02.112 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:51:02.234 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:51:02.236 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:51:02.324 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:51:02.325 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:51:02.427 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:51:02.569 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:51:02.657 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:51:02.659 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:51:02.782 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:51:02.783 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19581.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:51:02.853 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:51:02.992 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:51:03.049 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:51:03.051 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:51:03.198 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:51:03.199 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:51:03.240 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:51:03.402 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:51:03.441 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:51:03.444 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:51:03.613 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:51:03.614 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:51:03.655 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:51:03.827 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:51:03 CST 2025] <<<<<< run test case canal_json_claim_check success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:51:03.855 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:51:03.856 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:51:04.035 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:51:04.036 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:51:04.059 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] + set +x + tso='457114402880749569 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114402880749569 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114402880749569 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:51:04 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1960919611.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:51:04.240 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:51:04.250 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:51:04.252 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:51:04.441 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:51:04.442 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:51:04.443 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] check_redo_resolved_ts 8190197b-49ea-4b91-a651-d9b42f44d140 457114403054551043 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/8190197b-49ea-4b91-a651-d9b42f44d140/meta global resolved ts 457114402477834241 not forward to 457114403054551043 run task failed 1-th time, retry later [Fri Apr 4 15:51:04 CST 2025] <<<<<< START cdc server in safe_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.safe_mode.2144821450.out server --log-file /tmp/tidb_cdc_test/safe_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/safe_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:51:04.671 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:51:04.672 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:51:04.674 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:51:04.866 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:51:04.868 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:51:04.869 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:51:05.156 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:51:05.157 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:51:05.158 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:51:05.352 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:51:05.356 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:51:05.357 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] check_redo_resolved_ts 8190197b-49ea-4b91-a651-d9b42f44d140 457114403054551043 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/8190197b-49ea-4b91-a651-d9b42f44d140/meta global resolved ts 457114403002122242 not forward to 457114403054551043 run task failed 2-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:07 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97e73a86-e202-41a4-8917-1812150f02f9 {"id":"97e73a86-e202-41a4-8917-1812150f02f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7fb3ce1 97e73a86-e202-41a4-8917-1812150f02f9 /tidb/cdc/default/default/upstream/7489362332927618167 {"id":7489362332927618167,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97e73a86-e202-41a4-8917-1812150f02f9 {"id":"97e73a86-e202-41a4-8917-1812150f02f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7fb3ce1 97e73a86-e202-41a4-8917-1812150f02f9 /tidb/cdc/default/default/upstream/7489362332927618167 {"id":7489362332927618167,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97e73a86-e202-41a4-8917-1812150f02f9 {"id":"97e73a86-e202-41a4-8917-1812150f02f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7fb3ce1 97e73a86-e202-41a4-8917-1812150f02f9 /tidb/cdc/default/default/upstream/7489362332927618167 {"id":7489362332927618167,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114402880749569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19668.out cli changefeed create --start-ts=457114402880749569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362332927618167,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:51:07.662696036+08:00","start_ts":457114402880749569,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114402880749569,"checkpoint_ts":457114402880749569,"checkpoint_time":"2025-04-04 15:51:02.747"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:07 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ac9788c-d6bd-4a84-8834-7cdbb1eef897 {"id":"0ac9788c-d6bd-4a84-8834-7cdbb1eef897","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc803e6e8 0ac9788c-d6bd-4a84-8834-7cdbb1eef897 /tidb/cdc/default/default/upstream/7489362342483942439 {"id":7489362342483942439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ac9788c-d6bd-4a84-8834-7cdbb1eef897 {"id":"0ac9788c-d6bd-4a84-8834-7cdbb1eef897","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc803e6e8 0ac9788c-d6bd-4a84-8834-7cdbb1eef897 /tidb/cdc/default/default/upstream/7489362342483942439 {"id":7489362342483942439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ac9788c-d6bd-4a84-8834-7cdbb1eef897 {"id":"0ac9788c-d6bd-4a84-8834-7cdbb1eef897","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753064} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc803e6e8 0ac9788c-d6bd-4a84-8834-7cdbb1eef897 /tidb/cdc/default/default/upstream/7489362342483942439 {"id":7489362342483942439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.safe_mode.cli.21506.out cli changefeed create '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?safe-mode=true' Create changefeed successfully! ID: be436005-92e6-4b6a-8a3b-c1ea6775f1f3 Info: {"upstream_id":7489362342483942439,"namespace":"default","id":"be436005-92e6-4b6a-8a3b-c1ea6775f1f3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:51:07.821791116+08:00","start_ts":457114404182818817,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114404182818817,"checkpoint_ts":457114404182818817,"checkpoint_time":"2025-04-04 15:51:07.714"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:51:07 CST 2025] <<<<<< run test case open_protocol_claim_check success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + set +x Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_table_exists test.t1 127.0.0.1 3306 table test.t1 exists + sleep 5 check_redo_resolved_ts 8190197b-49ea-4b91-a651-d9b42f44d140 457114403054551043 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/8190197b-49ea-4b91-a651-d9b42f44d140/meta run task successfully wait process cdc.test exit for 1-th time... [2025/04/04 15:51:10.499 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] wait process cdc.test exit for 2-th time... [2025/04/04 15:51:10.773 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:51:10.780 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:51:10.990 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:51:10.991 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:51:11.010 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:51:11.013 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... [2025/04/04 15:51:11.228 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:51:11.249 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:51:11.462 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:51:11.463 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:51:11.493 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:51:11.495 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:51:11.707 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:51:11.741 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2025/04/04 15:51:11.872 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:51:11.873 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:51:11.873 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:51:11.874 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:51:11.878 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:51:11.878 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114397601955868] [resolvedTs=457114404050960386] [2025/04/04 15:51:11.878 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:51:11.879 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:51:11.879 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:51:11.879 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:51:11.879 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:51:11.882 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.768505ms] [2025/04/04 15:51:11.886 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:51:11.890 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.891 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.892 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.892 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.892 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:51:11.892 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:51:11.892 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:51:11.892 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:51:11.892 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:51:11.900 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:51:11.903 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:51:11.903 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:51:11.982 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:51:11.984 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:51:11 CST 2025] <<<<<< run test case canal_json_storage_basic success! >>>>>> [2025/04/04 15:51:12.182 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:51:12.186 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:51:12.285 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:51:12.534 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:51:12.535 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:51:12.599 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:51:12.823 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:51:12.991 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:51:12.995 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:51:13.147 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:51:13.148 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:51:13.396 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:51:13.477 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362393721772314 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2129e4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:24676, start at 2025-04-04 15:51:11.240417408 +0800 CST m=+2.100811277 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:11.250 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:11.256 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:11.256 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362393721772314 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2129e4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:24676, start at 2025-04-04 15:51:11.240417408 +0800 CST m=+2.100811277 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:11.250 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:11.256 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:11.256 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362391719834082 TiDB Cluster ID. tikv_gc_leader_uuid 657ff212a680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-23b4k-7ks82-5s2w3, pid:24735, start at 2025-04-04 15:51:11.264160261 +0800 CST m=+2.034904915 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:11.271 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:11.258 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:11.258 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:51:13.789 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:51:13.793 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:51:13.814 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:51:13.816 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:51:14.167 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:51:13.868 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.990387379s] [2025/04/04 15:51:13.869 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114397601955863,\"CommitTs\":457114397601955868,\"Query\":\"ALTER TABLE `consistent_replicate_storage_s3`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_s3`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_s3\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:51:14.256 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:51:14.504 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:51:14.505 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:51:14.706 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:51:14.710 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17607 23.8 0.6 4854260 2497064 ? Sl 15:50 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 17608 31.0 0.6 4893688 2538324 ? Sl 15:50 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 17609 23.4 0.6 4845044 2485832 ? Sl 15:50 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 17616 30.5 0.6 4877296 2525064 ? Sl 15:50 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 17357 0 --:--:-- --:--:-- --:--:-- 17357 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:51:14.147","puller_resolved_ts":"2025-04-04 15:51:09.097","last_synced_ts":"2025-04-04 15:51:09.597","now_ts":"2025-04-04 15:51:14.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:14.147","puller_resolved_ts":"2025-04-04' '15:51:09.097","last_synced_ts":"2025-04-04' '15:51:09.597","now_ts":"2025-04-04' '15:51:14.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:14.147","puller_resolved_ts":"2025-04-04' '15:51:09.097","last_synced_ts":"2025-04-04' '15:51:09.597","now_ts":"2025-04-04' '15:51:14.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 [2025/04/04 15:51:14.884 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:51:15.066 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:51:15.204 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:51:15.205 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:51:15.385 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:51:15.390 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.26057.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:51:15.528 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:15.529 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:15.629 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:15.629 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:16.131 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:16.131 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114404050960386] [2025/04/04 15:51:16.231 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=457114404050960386] [2025/04/04 15:51:16.231 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:51:16.231 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:51:16.231 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:51:16.232 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:51:16.231 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] Apply redo log successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:51:16 CST 2025] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> check diff successfully [Fri Apr 4 15:51:17 CST 2025] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> count(*) 5000 Exiting on signal: INTERRUPT + set +x + tso='457114406317457409 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114406317457409 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:51:18 CST 2025] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.2610026102.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:21 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d1efe14-3dce-4c8f-9fec-8d329fd5f27f {"id":"2d1efe14-3dce-4c8f-9fec-8d329fd5f27f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753079} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc82e4309 2d1efe14-3dce-4c8f-9fec-8d329fd5f27f /tidb/cdc/default/default/upstream/7489362393721772314 {"id":7489362393721772314,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d1efe14-3dce-4c8f-9fec-8d329fd5f27f {"id":"2d1efe14-3dce-4c8f-9fec-8d329fd5f27f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753079} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc82e4309 2d1efe14-3dce-4c8f-9fec-8d329fd5f27f /tidb/cdc/default/default/upstream/7489362393721772314 {"id":7489362393721772314,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d1efe14-3dce-4c8f-9fec-8d329fd5f27f {"id":"2d1efe14-3dce-4c8f-9fec-8d329fd5f27f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753079} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc82e4309 2d1efe14-3dce-4c8f-9fec-8d329fd5f27f /tidb/cdc/default/default/upstream/7489362393721772314 {"id":7489362393721772314,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.26166.out cli changefeed create --start-ts=457114406317457409 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml Create changefeed successfully! ID: 48056e29-a1a6-406b-8012-278c70bd233a Info: {"upstream_id":7489362393721772314,"namespace":"default","id":"48056e29-a1a6-406b-8012-278c70bd233a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:51:22.407153465+08:00","start_ts":457114406317457409,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114406317457409,"checkpoint_ts":457114406317457409,"checkpoint_time":"2025-04-04 15:51:15.857"} PASS * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330E309B7ED9E8 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:51:22 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact coverage: 2.4% of statements in github.com/pingcap/tiflow/... Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... + set +x table split_region.test1 not exists for 1-th check, retry later [2025/04/04 15:51:24.631 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m4.973059313s"] table mark.finish_mark_0 exists table mark.finish_mark_1 not exists for 1-th check, retry later table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later table mark.finish_mark_1 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/multi_tables_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table split_region.test2 exists check diff successfully Verifying downstream PD is started... table mark.finish_mark_1 not exists for 3-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/consistent_partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table mark.finish_mark_1 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table mark.finish_mark_1 not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362484009161805 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2294080002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:45225, start at 2025-04-04 15:51:34.426954633 +0800 CST m=+2.252800365 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:34.436 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:34.402 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:34.402 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362484009161805 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2294080002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:45225, start at 2025-04-04 15:51:34.426954633 +0800 CST m=+2.252800365 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:34.436 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:34.402 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:34.402 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362491317755384 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2295940001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-csk1g-gh5zq-t0xpd, pid:45290, start at 2025-04-04 15:51:34.525641397 +0800 CST m=+2.276515009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:34.533 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:34.501 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:34.501 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_1 not exists for 7-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362495438263698 TiDB Cluster ID. tikv_gc_leader_uuid 657ff22ac9c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:18837, start at 2025-04-04 15:51:36.017492509 +0800 CST m=+2.062595059 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:36.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:36.024 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:36.024 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362495438263698 TiDB Cluster ID. tikv_gc_leader_uuid 657ff22ac9c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:18837, start at 2025-04-04 15:51:36.017492509 +0800 CST m=+2.062595059 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:36.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:36.024 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:36.024 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362497701147805 TiDB Cluster ID. tikv_gc_leader_uuid 657ff22aee4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-d3bms-87pgm-38mp6, pid:18914, start at 2025-04-04 15:51:36.14687076 +0800 CST m=+2.113919093 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:36.159 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:36.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:36.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:51:38 CST 2025] <<<<<< START cdc server in multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_tables_ddl.4669546697.out server --log-file /tmp/tidb_cdc_test/multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_tables_ddl/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_1 not exists for 8-th check, retry later check diff failed 1-th time, retry later [Fri Apr 4 15:51:40 CST 2025] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2015620158.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_1 not exists for 9-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd1d39d8-2b0c-4064-8bf9-69434a785236 {"id":"cd1d39d8-2b0c-4064-8bf9-69434a785236","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753099} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc88a09db cd1d39d8-2b0c-4064-8bf9-69434a785236 /tidb/cdc/default/default/upstream/7489362484009161805 {"id":7489362484009161805,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd1d39d8-2b0c-4064-8bf9-69434a785236 {"id":"cd1d39d8-2b0c-4064-8bf9-69434a785236","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753099} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc88a09db cd1d39d8-2b0c-4064-8bf9-69434a785236 /tidb/cdc/default/default/upstream/7489362484009161805 {"id":7489362484009161805,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd1d39d8-2b0c-4064-8bf9-69434a785236 {"id":"cd1d39d8-2b0c-4064-8bf9-69434a785236","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753099} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc88a09db cd1d39d8-2b0c-4064-8bf9-69434a785236 /tidb/cdc/default/default/upstream/7489362484009161805 {"id":7489362484009161805,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7489362484009161805,"namespace":"default","id":"test-normal","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:51:41.900608953+08:00","start_ts":457114412293292033,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114412293292033,"checkpoint_ts":457114412293292033,"checkpoint_time":"2025-04-04 15:51:38.653"} Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7489362484009161805,"namespace":"default","id":"test-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:51:42.033299486+08:00","start_ts":457114412293292033,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114412293292033,"checkpoint_ts":457114412293292033,"checkpoint_time":"2025-04-04 15:51:38.653"} check diff failed 2-th time, retry later Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7489362484009161805,"namespace":"default","id":"test-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:51:42.201787616+08:00","start_ts":457114412293292033,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114412293292033,"checkpoint_ts":457114412293292033,"checkpoint_time":"2025-04-04 15:51:38.653"} table mark.finish_mark_1 not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:43 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 {"id":"7d654cd7-b8d6-46d6-a165-cbc8f1def7e7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8950ed2 7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 {"id":"7d654cd7-b8d6-46d6-a165-cbc8f1def7e7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8950ed2 7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 {"id":"7d654cd7-b8d6-46d6-a165-cbc8f1def7e7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8950ed2 7d654cd7-b8d6-46d6-a165-cbc8f1def7e7 /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully wait process cdc.test exit for 1-th time... table mark.finish_mark_1 not exists for 11-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:51:45 CST 2025] <<<<<< run test case split_region success! >>>>>> table mark.finish_mark_1 not exists for 12-th check, retry later table partition_table2.t2 not exists for 1-th check, retry later table mark.finish_mark_1 not exists for 13-th check, retry later table partition_table2.t2 not exists for 2-th check, retry later table mark.finish_mark_1 not exists for 14-th check, retry later table partition_table2.t2 not exists for 3-th check, retry later table mark.finish_mark_1 not exists for 15-th check, retry later table partition_table2.t2 not exists for 4-th check, retry later table mark.finish_mark_1 not exists for 16-th check, retry later table partition_table2.t2 not exists for 5-th check, retry later table safe_mode.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:51:50 CST 2025] <<<<<< run test case safe_mode success! >>>>>> table mark.finish_mark_1 not exists for 17-th check, retry later table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later table partition_table2.t2 not exists for 6-th check, retry later table multi_tables_ddl_test.finish_mark exists check table exists success \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } table mark.finish_mark_1 not exists for 18-th check, retry later [Pipeline] // withEnv [Pipeline] } + info='{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114416697311256, "checkpoint_time": "2025-04-04 15:51:55.453", "error": null }' + echo '{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114416697311256, "checkpoint_time": "2025-04-04 15:51:55.453", "error": null }' { "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114416697311256, "checkpoint_time": "2025-04-04 15:51:55.453", "error": null } ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114416697311256, '"checkpoint_time":' '"2025-04-04' '15:51:55.453",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114416697311256, '"checkpoint_time":' '"2025-04-04' '15:51:55.453",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s + info='{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114417431314437, "checkpoint_time": "2025-04-04 15:51:58.253", "error": null }' + echo '{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114417431314437, "checkpoint_time": "2025-04-04 15:51:58.253", "error": null }' { "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114417431314437, "checkpoint_time": "2025-04-04 15:51:58.253", "error": null } ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114417431314437, '"checkpoint_time":' '"2025-04-04' '15:51:58.253",' '"error":' null '}' ++ jq -r .state [Pipeline] // stage [Pipeline] } + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114417431314437, '"checkpoint_time":' '"2025-04-04' '15:51:58.253",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s + info='{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114416382738435, "checkpoint_time": "2025-04-04 15:51:54.253", "error": { "time": "2025-04-04T15:51:55.283357741+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114416382738435, "checkpoint_time": "2025-04-04 15:51:54.253", "error": { "time": "2025-04-04T15:51:55.283357741+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7489362484009161805, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114416382738435, "checkpoint_time": "2025-04-04 15:51:54.253", "error": { "time": "2025-04-04T15:51:55.283357741+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114416382738435, '"checkpoint_time":' '"2025-04-04' '15:51:54.253",' '"error":' '{' '"time":' '"2025-04-04T15:51:55.283357741+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489362484009161805, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114416382738435, '"checkpoint_time":' '"2025-04-04' '15:51:54.253",' '"error":' '{' '"time":' '"2025-04-04T15:51:55.283357741+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully table partition_table2.t2 not exists for 7-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:52:00 CST 2025] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:00 CST 2025] <<<<<< run test case kafka_big_messages success! >>>>>> table mark.finish_mark_1 not exists for 19-th check, retry later table partition_table2.t2 not exists for 8-th check, retry later table mark.finish_mark_1 not exists for 20-th check, retry later table partition_table2.t2 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:03 CST 2025] <<<<<< run test case kafka_compression success! >>>>>> table mark.finish_mark_1 not exists for 21-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:06 CST 2025] <<<<<< run test case kafka_messages success! >>>>>> table mark.finish_mark_1 not exists for 22-th check, retry later table mark.finish_mark_1 not exists for 23-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:52:09 CST 2025] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2032320325.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server2 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:09 CST 2025] <<<<<< run test case kafka_sink_error_resume success! >>>>>> table mark.finish_mark_1 not exists for 24-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:52:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/9a40b058-7669-4ac9-9018-cf01c490c0d1 {UpstreamID:7489362495438263698 Namespace:default ID:9a40b058-7669-4ac9-9018-cf01c490c0d1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:51:43.615737635 +0800 CST StartTs:457114413570195463 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00415c900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114413583564808} {CheckpointTs:457114419730579461 MinTableBarrierTs:457114421041299472 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aabaade-d820-4b45-a56d-22fcee77fdf1 {"id":"4aabaade-d820-4b45-a56d-22fcee77fdf1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753130} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8951034 4aabaade-d820-4b45-a56d-22fcee77fdf1 /tidb/cdc/default/default/changefeed/info/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"upstream-id":7489362495438263698,"namespace":"default","changefeed-id":"9a40b058-7669-4ac9-9018-cf01c490c0d1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:51:43.615737635+08:00","start-ts":457114413570195463,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114413583564808} /tidb/cdc/default/default/changefeed/status/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":457114419730579461,"min-table-barrier-ts":457114421041299472,"admin-job-type":0} /tidb/cdc/default/default/task/position/4aabaade-d820-4b45-a56d-22fcee77fdf1/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9a40b058-7669-4ac9-9018-cf01c490c0d1 {UpstreamID:7489362495438263698 Namespace:default ID:9a40b058-7669-4ac9-9018-cf01c490c0d1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:51:43.615737635 +0800 CST StartTs:457114413570195463 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00415c900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114413583564808} {CheckpointTs:457114419730579461 MinTableBarrierTs:457114421041299472 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aabaade-d820-4b45-a56d-22fcee77fdf1 {"id":"4aabaade-d820-4b45-a56d-22fcee77fdf1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753130} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8951034 4aabaade-d820-4b45-a56d-22fcee77fdf1 /tidb/cdc/default/default/changefeed/info/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"upstream-id":7489362495438263698,"namespace":"default","changefeed-id":"9a40b058-7669-4ac9-9018-cf01c490c0d1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:51:43.615737635+08:00","start-ts":457114413570195463,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114413583564808} /tidb/cdc/default/default/changefeed/status/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":457114419730579461,"min-table-barrier-ts":457114421041299472,"admin-job-type":0} /tidb/cdc/default/default/task/position/4aabaade-d820-4b45-a56d-22fcee77fdf1/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9a40b058-7669-4ac9-9018-cf01c490c0d1 {UpstreamID:7489362495438263698 Namespace:default ID:9a40b058-7669-4ac9-9018-cf01c490c0d1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:51:43.615737635 +0800 CST StartTs:457114413570195463 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00415c900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114413583564808} {CheckpointTs:457114419730579461 MinTableBarrierTs:457114421041299472 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114419730579461, checkpointTs: 457114419730579461, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4aabaade-d820-4b45-a56d-22fcee77fdf1 {"id":"4aabaade-d820-4b45-a56d-22fcee77fdf1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753130} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8951034 4aabaade-d820-4b45-a56d-22fcee77fdf1 /tidb/cdc/default/default/changefeed/info/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"upstream-id":7489362495438263698,"namespace":"default","changefeed-id":"9a40b058-7669-4ac9-9018-cf01c490c0d1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:51:43.615737635+08:00","start-ts":457114413570195463,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-inelig+ grep -q 'etcd info' ible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114413583564808} /tidb/cdc/default/default/changefeed/status/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":457114419730579461,"min-table-barrier-ts":457114421041299472,"admin-job-type":0} /tidb/cdc/default/default/task/position/4aabaade-d820-4b45-a56d-22fcee77fdf1/9a40b058-7669-4ac9-9018-cf01c490c0d1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489362495438263698 {"id":7489362495438263698,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:13 CST 2025] <<<<<< run test case mq_sink_lost_callback success! >>>>>> table mark.finish_mark_1 not exists for 25-th check, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table mark.finish_mark_1 not exists for 26-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:16 CST 2025] <<<<<< run test case mq_sink_dispatcher success! >>>>>> table mark.finish_mark_1 not exists for 27-th check, retry later table mark.finish_mark_1 not exists for 28-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:19 CST 2025] <<<<<< run test case kafka_column_selector success! >>>>>> table mark.finish_mark_1 not exists for 29-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:22 CST 2025] <<<<<< run test case kafka_column_selector_avro success! >>>>>> table mark.finish_mark_1 not exists for 30-th check, retry later table mark.finish_mark_1 not exists for 31-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: mysql... <<================= =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:25 CST 2025] <<<<<< run test case lossy_ddl success! >>>>>> table mark.finish_mark_1 exists table mark.finish_mark_2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:28 CST 2025] <<<<<< run test case storage_csv_update success! >>>>>> table mark.finish_mark_2 not exists for 2-th check, retry later table mark.finish_mark_2 not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:31 CST 2025] <<<<<< run test case csv_storage_update_pk_clustered success! >>>>>> table mark.finish_mark_2 not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:52:35 CST 2025] <<<<<< run test case csv_storage_update_pk_nonclustered success! >>>>>> table mark.finish_mark_2 not exists for 5-th check, retry later table mark.finish_mark_2 not exists for 6-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table mark.finish_mark_2 not exists for 7-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table mark.finish_mark_2 not exists for 8-th check, retry later table mark.finish_mark_2 not exists for 9-th check, retry later table mark.finish_mark_2 exists table mark.finish_mark_3 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 3-th check, retry later table mark.finish_mark_3 not exists for 4-th check, retry later table mark.finish_mark_3 not exists for 5-th check, retry later table mark.finish_mark_3 not exists for 6-th check, retry later table mark.finish_mark_3 not exists for 7-th check, retry later table mark.finish_mark_3 not exists for 8-th check, retry later table mark.finish_mark_3 exists table mark.finish_mark_4 not exists for 1-th check, retry later table mark.finish_mark_4 not exists for 2-th check, retry later table mark.finish_mark_4 not exists for 3-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2728 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:52:54.677","puller_resolved_ts":"2025-04-04 15:52:48.677","last_synced_ts":"2025-04-04 15:50:38.476","now_ts":"2025-04-04 15:52:55.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:52:54.677","puller_resolved_ts":"2025-04-04' '15:52:48.677","last_synced_ts":"2025-04-04' '15:50:38.476","now_ts":"2025-04-04' '15:52:55.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:52:54.677","puller_resolved_ts":"2025-04-04' '15:52:48.677","last_synced_ts":"2025-04-04' '15:50:38.476","now_ts":"2025-04-04' '15:52:55.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory table mark.finish_mark_4 not exists for 4-th check, retry later chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table mark.finish_mark_4 not exists for 5-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table mark.finish_mark_4 not exists for 6-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_4 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_4 not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362920219505406 TiDB Cluster ID. tikv_gc_leader_uuid 657ff28ba440001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:17092, start at 2025-04-04 15:53:15.153646955 +0800 CST m=+2.274798209 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:15.162 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:15.167 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:15.167 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362920219505406 TiDB Cluster ID. tikv_gc_leader_uuid 657ff28ba440001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:17092, start at 2025-04-04 15:53:15.153646955 +0800 CST m=+2.274798209 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:15.162 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:15.167 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:15.167 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362920114254167 TiDB Cluster ID. tikv_gc_leader_uuid 657ff28ba58000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-t3v13-m4jvc-3dq32, pid:17152, start at 2025-04-04 15:53:15.202082796 +0800 CST m=+2.231479029 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:15.209 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:15.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:15.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log table mark.finish_mark_4 exists table mark.finish_mark_5 not exists for 1-th check, retry later arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_5 not exists for 2-th check, retry later + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18511.out cli tso query --pd=http://127.0.0.1:2379 table mark.finish_mark_5 not exists for 3-th check, retry later + set +x + tso='457114438747553793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114438747553793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114438747553793 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:53:20 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1854318545.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_5 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:53:24 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e7047cb5-5634-4e0e-895e-9139e34f1135 {"id":"e7047cb5-5634-4e0e-895e-9139e34f1135","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753201} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca11b1e2 e7047cb5-5634-4e0e-895e-9139e34f1135 /tidb/cdc/default/default/upstream/7489362920219505406 {"id":7489362920219505406,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e7047cb5-5634-4e0e-895e-9139e34f1135 {"id":"e7047cb5-5634-4e0e-895e-9139e34f1135","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753201} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca11b1e2 e7047cb5-5634-4e0e-895e-9139e34f1135 /tidb/cdc/default/default/upstream/7489362920219505406 {"id":7489362920219505406,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e7047cb5-5634-4e0e-895e-9139e34f1135 {"id":"e7047cb5-5634-4e0e-895e-9139e34f1135","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753201} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca11b1e2 e7047cb5-5634-4e0e-895e-9139e34f1135 /tidb/cdc/default/default/upstream/7489362920219505406 {"id":7489362920219505406,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114438747553793 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18594.out cli changefeed create --start-ts=457114438747553793 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362920219505406,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:53:24.420691137+08:00","start_ts":457114438747553793,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114438747553793,"checkpoint_ts":457114438747553793,"checkpoint_time":"2025-04-04 15:53:19.568"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_5 not exists for 5-th check, retry later + set +x + sleep 20 table mark.finish_mark_5 not exists for 6-th check, retry later table mark.finish_mark_5 not exists for 7-th check, retry later table mark.finish_mark_5 exists table mark.finish_mark not exists for 1-th check, retry later table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:53:35 CST 2025] <<<<<< run test case multi_source success! >>>>>> ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 23322 0 --:--:-- --:--:-- --:--:-- 23322 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:51:14.147","puller_resolved_ts":"2025-04-04 15:51:14.147","last_synced_ts":"2025-04-04 15:51:09.597","now_ts":"2025-04-04 15:53:24.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:14.147","puller_resolved_ts":"2025-04-04' '15:51:14.147","last_synced_ts":"2025-04-04' '15:51:09.597","now_ts":"2025-04-04' '15:53:24.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:14.147","puller_resolved_ts":"2025-04-04' '15:51:14.147","last_synced_ts":"2025-04-04' '15:51:09.597","now_ts":"2025-04-04' '15:53:24.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/vector/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363040048146209 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2a94380006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:20743, start at 2025-04-04 15:53:45.493540723 +0800 CST m=+2.006984193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:45.538 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:45.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:45.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 28920 0 --:--:-- --:--:-- --:--:-- 28920 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:53:19.568","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:53:45.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:19.568","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:53:45.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:19.568","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:53:45.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363040048146209 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2a94380006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:20743, start at 2025-04-04 15:53:45.493540723 +0800 CST m=+2.006984193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:45.538 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:45.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:45.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363054182636767 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2a9684000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:20824, start at 2025-04-04 15:53:45.677300798 +0800 CST m=+2.117448064 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:45.684 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:45.684 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:45.684 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/vector Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22201.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114446774140929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114446774140929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114446774140929 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:53:51 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2223822240.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:53:54 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15d00c95-a922-4b3e-9ca1-79943614477e {"id":"15d00c95-a922-4b3e-9ca1-79943614477e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8533ec 15d00c95-a922-4b3e-9ca1-79943614477e /tidb/cdc/default/default/upstream/7489363040048146209 {"id":7489363040048146209,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15d00c95-a922-4b3e-9ca1-79943614477e {"id":"15d00c95-a922-4b3e-9ca1-79943614477e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8533ec 15d00c95-a922-4b3e-9ca1-79943614477e /tidb/cdc/default/default/upstream/7489363040048146209 {"id":7489363040048146209,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15d00c95-a922-4b3e-9ca1-79943614477e {"id":"15d00c95-a922-4b3e-9ca1-79943614477e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8533ec 15d00c95-a922-4b3e-9ca1-79943614477e /tidb/cdc/default/default/upstream/7489363040048146209 {"id":7489363040048146209,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114446774140929 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22293.out cli changefeed create --start-ts=457114446774140929 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489363040048146209,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:53:55.067086046+08:00","start_ts":457114446774140929,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114446774140929,"checkpoint_ts":457114446774140929,"checkpoint_time":"2025-04-04 15:53:50.187"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Fri Apr 4 15:53:56 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Fri Apr 4 15:53:56 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363099667447850 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2b3e540014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:29676, start at 2025-04-04 15:53:56.404784066 +0800 CST m=+2.217520536 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:56.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:56.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:56.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363099667447850 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2b3e540014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:29676, start at 2025-04-04 15:53:56.404784066 +0800 CST m=+2.217520536 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:56.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:56.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:56.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363096273263145 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2b4150000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-l1r4j-cwb9x-9b2xs, pid:29752, start at 2025-04-04 15:53:56.610228572 +0800 CST m=+2.335710619 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:56.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:56.615 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:56.615 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table test.t1 exists + sleep 5 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/vector/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/vector/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/vector/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/vector/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/vector/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.31058.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + tso='457114449575411713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114449575411713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:54:02 CST 2025] <<<<<< START cdc server in vector case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.3109231094.out server --log-file /tmp/tidb_cdc_test/vector/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/vector/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20743 7.7 0.0 3184444 218148 ? Sl 15:53 0:01 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753223476251041.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 20752 3.5 0.0 3404968 187620 ? Sl 15:53 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753223484286339.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 20824 9.4 0.0 3332420 215464 ? Sl 15:53 0:01 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753223548401778.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 48600 0 --:--:-- --:--:-- --:--:-- 48600 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:54:02.537","puller_resolved_ts":"2025-04-04 15:53:56.487","last_synced_ts":"2025-04-04 15:53:56.537","now_ts":"2025-04-04 15:54:03.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:54:02.537","puller_resolved_ts":"2025-04-04' '15:53:56.487","last_synced_ts":"2025-04-04' '15:53:56.537","now_ts":"2025-04-04' '15:54:03.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:54:02.537","puller_resolved_ts":"2025-04-04' '15:53:56.487","last_synced_ts":"2025-04-04' '15:53:56.537","now_ts":"2025-04-04' '15:54:03.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:54:05 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1a3523b-d917-4e31-aafb-5309e28704f6 {"id":"d1a3523b-d917-4e31-aafb-5309e28704f6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcaaf21e5 d1a3523b-d917-4e31-aafb-5309e28704f6 /tidb/cdc/default/default/upstream/7489363099667447850 {"id":7489363099667447850,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1a3523b-d917-4e31-aafb-5309e28704f6 {"id":"d1a3523b-d917-4e31-aafb-5309e28704f6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcaaf21e5 d1a3523b-d917-4e31-aafb-5309e28704f6 /tidb/cdc/default/default/upstream/7489363099667447850 {"id":7489363099667447850,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1a3523b-d917-4e31-aafb-5309e28704f6 {"id":"d1a3523b-d917-4e31-aafb-5309e28704f6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcaaf21e5 d1a3523b-d917-4e31-aafb-5309e28704f6 /tidb/cdc/default/default/upstream/7489363099667447850 {"id":7489363099667447850,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.31142.out cli changefeed create --start-ts=457114449575411713 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 2614e089-c14e-450f-bc1b-1937db67ec31 Info: {"upstream_id":7489363099667447850,"namespace":"default","id":"2614e089-c14e-450f-bc1b-1937db67ec31","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:54:05.857947407+08:00","start_ts":457114449575411713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114449575411713,"checkpoint_ts":457114449575411713,"checkpoint_time":"2025-04-04 15:54:00.873"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:54:16 CST 2025] <<<<<< run test case vector success! >>>>>> wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/partition_table/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["partition_table.?*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:54:18.229 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:54:18.230 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:54:18.230 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:54:18.230 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:54:18.230 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114419730579461] [resolvedTs=457114453586477060] [2025/04/04 15:54:18.231 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:54:18.231 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:54:18.231 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:54:18.232 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.312152ms] [2025/04/04 15:54:18.232 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.587457ms] [2025/04/04 15:54:18.235 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:54:18.238 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:54:18.239 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:54:18.239 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:54:18.239 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:54:18.239 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:54:18.243 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:54:18.245 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:54:18.245 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:54:18.245 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114421290336304,\"CommitTs\":457114421290336307,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:54:20.253 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:54:20.253 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114421290336307] [DDL="alter table t1 drop partition p1"] [2025/04/04 15:54:20.305 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=51.654384ms] [sql="alter table t1 drop partition p1"] [2025/04/04 15:54:20.305 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114421421408282,\"CommitTs\":457114421421408287,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:54:20.505 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114421421408287] [DDL="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [2025/04/04 15:54:20.539 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=34.405124ms] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [2025/04/04 15:54:20.539 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114421552742430,\"CommitTs\":457114421552742437,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:54:20.941 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114421552742437] [DDL="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [2025/04/04 15:54:20.974 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=32.817932ms] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [2025/04/04 15:54:20.974 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114421958803480,\"CommitTs\":457114421958803484,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:54:21.374 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114421958803484] [DDL="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [2025/04/04 15:54:21.858 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=483.69686ms] [sql="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [2025/04/04 15:54:21.858 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:21.858 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:21.958 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:21.958 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.058 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.058 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.159 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.159 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.259 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.259 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.359 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.359 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.459 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.459 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.559 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.560 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.660 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.660 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.760 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.760 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.860 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.860 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.960 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:22.960 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}"] [checkpointTs=457114453586477060] [2025/04/04 15:54:23.060 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=47] [appliedDDLCount=4] [currentCheckpoint=457114453586477060] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:54:23.061 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:54:23.061 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:54:23.060 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:54:23.061 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:54:23.061 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] Apply redo log successfully check diff successfully [Fri Apr 4 15:54:23 CST 2025] <<<<<< run test case consistent_partition_table success! >>>>>> Database INFORMATION_SCHEMA METRICS_SCHEMA PERFORMANCE_SCHEMA mysql partition_table partition_table2 sys test \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:32 CST 2025] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:36 CST 2025] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:39 CST 2025] <<<<<< run test case multi_topics_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=mysql + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run mysql + '[' mysql '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 4 15:54:42 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 4 15:54:42 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:46 CST 2025] <<<<<< run test case csv_storage_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:49 CST 2025] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:54:53 CST 2025] <<<<<< run test case csv_storage_partition_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7366 0 --:--:-- --:--:-- --:--:-- 7366 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:56:12.687","puller_resolved_ts":"2025-04-04 15:56:05.687","last_synced_ts":"2025-04-04 15:53:56.537","now_ts":"2025-04-04 15:56:13.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:56:12.687","puller_resolved_ts":"2025-04-04' '15:56:05.687","last_synced_ts":"2025-04-04' '15:53:56.537","now_ts":"2025-04-04' '15:56:13.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:56:12.687","puller_resolved_ts":"2025-04-04' '15:56:05.687","last_synced_ts":"2025-04-04' '15:53:56.537","now_ts":"2025-04-04' '15:56:13.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363773066006877 TiDB Cluster ID. tikv_gc_leader_uuid 657ff34dc4c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:23434, start at 2025-04-04 15:56:33.943897811 +0800 CST m=+2.103294442 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:58:33.951 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:56:33.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:46:33.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363773066006877 TiDB Cluster ID. tikv_gc_leader_uuid 657ff34dc4c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:23434, start at 2025-04-04 15:56:33.943897811 +0800 CST m=+2.103294442 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:58:33.951 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:56:33.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:46:33.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363776986204412 TiDB Cluster ID. tikv_gc_leader_uuid 657ff34dc3c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3828-7dz70-dw7fn-04n77, pid:23517, start at 2025-04-04 15:56:33.974387072 +0800 CST m=+2.031645955 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:58:33.981 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:56:33.985 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:46:33.985 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24837.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114490906607618 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114490906607618 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114490906607618 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:56:39 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2487024872.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:56:43 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 {"id":"3dc495d8-1c69-4fe5-9bc4-1b6200f41d19","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753400} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcd18c7e9 3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 /tidb/cdc/default/default/upstream/7489363773066006877 {"id":7489363773066006877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 {"id":"3dc495d8-1c69-4fe5-9bc4-1b6200f41d19","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753400} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcd18c7e9 3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 /tidb/cdc/default/default/upstream/7489363773066006877 {"id":7489363773066006877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 {"id":"3dc495d8-1c69-4fe5-9bc4-1b6200f41d19","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753400} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcd18c7e9 3dc495d8-1c69-4fe5-9bc4-1b6200f41d19 /tidb/cdc/default/default/upstream/7489363773066006877 {"id":7489363773066006877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114490906607618 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24924.out cli changefeed create --start-ts=457114490906607618 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489363773066006877,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:56:43.462602724+08:00","start_ts":457114490906607618,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114490906607618,"checkpoint_ts":457114490906607618,"checkpoint_time":"2025-04-04 15:56:38.539"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 42529 0 --:--:-- --:--:-- --:--:-- 42529 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:56:38.539","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:57:04.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:56:38.539","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:57:04.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:56:38.539","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:57:04.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 4 15:57:15 CST 2025] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 4 15:57:15 CST 2025] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3828/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS