Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/latest/pull_cdc_integration_storage_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-hnmwr-xz8nj --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e0084154211910533e6b09e86a3d7c29c4144c6a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-hnmwr" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git rev-list --no-walk b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} _=/usr/bin/printenv TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-hnmwr HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=31397bcc-01ec-4747-b025-4f201cfbe1ef GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Debug info JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=3 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-hnmwr pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-ec41a9ea2975eb575c6c655de75271b1a69674254216af41486e1697505d3c9f + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.23.3' GODEBUG='' GOTELEMETRY='local' GOTELEMETRYDIR='/home/jenkins/.config/go/telemetry' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3447129067=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs bash' debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-storage-test-3706-hnm-q38gs bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56014 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 3b979fe45574d3d4 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 27 IN A 20.205.243.166 github.com. 27 IN A 20.27.177.113 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 04 07:24:11 UTC 2025 ;; MSG SIZE rcvd: 103 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [.github/workflows/check_and_build.yaml, .github/workflows/dataflow_engine_chaos.yaml, .github/workflows/dm_binlog_999999.yaml, .github/workflows/dm_chaos.yaml, .github/workflows/dm_mariadb_master_down_and_up.yaml, .github/workflows/dm_upstream_switch.yaml] [Pipeline] echo diff file not matched: .github/workflows/check_and_build.yaml [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-4c631d5-e8bfbe4) 192358400 bytes in 0.83 secs (232472795 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.43.5 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/.git/ .git HEAD is now at e8bfbe4ba CI: Upgrade to actions/cache@v4 POST git-upload-pack (656 bytes) From https://github.com/pingcap/tiflow = [up to date] master -> origin/master = [up to date] refs/pull/12120/head -> origin/pr/12120/head Previous HEAD position was e8bfbe4ba CI: Upgrade to actions/cache@v4 HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) ๐Ÿšง Checkouting to base SHA:4c631d5951854ac91765b729fe3849d8e8d5ed8a... HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 4c631d5951854ac91765b729fe3849d8e8d5ed8a 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) 9ed3d76c5 consumer(ticdc): adjust consumer log level (#12092) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 4c631d5951854ac91765b729fe3849d8e8d5ed8a ... Updating 4c631d595..e8bfbe4ba Fast-forward .github/workflows/check_and_build.yaml | 2 +- .github/workflows/dataflow_engine_chaos.yaml | 2 +- .github/workflows/dm_binlog_999999.yaml | 4 ++-- .github/workflows/dm_chaos.yaml | 4 ++-- .github/workflows/dm_mariadb_master_down_and_up.yaml | 4 ++-- .github/workflows/dm_upstream_switch.yaml | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) ๐Ÿงพ Pre-merged result: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 e8bfbe4ba CI: Upgrade to actions/cache@v4 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-4c631d5-e8bfbe4 already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/third_party_download [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] echo computeBranchFromPR component: tidb, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: pd, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tikv, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tiflash, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] retry [Pipeline] { [Pipeline] sh + export TIDB_BRANCH=master + TIDB_BRANCH=master + export PD_BRANCH=master + PD_BRANCH=master + export TIKV_BRANCH=master + TIKV_BRANCH=master + export TIFLASH_BRANCH=master + TIFLASH_BRANCH=master + cd ../tiflow + ./scripts/download-integration-test-binaries.sh master Downloading binaries... >>> Downloading tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz 2025-04-04 15:24:30 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz [620275338/620275338] -> "tmp/tidb-server.tar.gz" [1] >>> Downloading pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz 2025-04-04 15:24:47 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz [228444265/228444265] -> "tmp/pd-server.tar.gz" [1] >>> Downloading tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz 2025-04-04 15:25:08 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz [817400808/817400808] -> "tmp/tikv-server.tar.gz" [1] >>> Downloading tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz 2025-04-04 15:25:33 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz [553235170/553235170] -> "tmp/tiflash.tar.gz" [1] >>> Downloading minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2025-04-04 15:25:39 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> Downloading etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2025-04-04 15:25:40 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> Downloading schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2025-04-04 15:25:49 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] >>> Downloading go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> Downloading jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] Download SUCCESS + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + make check_third_party_binary go: downloading go1.23.4 (linux/amd64) /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-region-bench ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/pd-ut ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ../tiflow/bin/xprog ./bin/ + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + ./bin/tidb-server -V Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + ./bin/tikv-server -V TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:59" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/sync v0.10.0 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/net v0.33.0 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/coreos/go-semver v0.3.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/robfig/cron v1.2.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.35.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/golang/protobuf v1.5.4 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.22.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading k8s.io/apimachinery v0.29.11 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:29:02" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer go: downloading github.com/confluentinc/confluent-kafka-go/v2 v2.3.0 go: downloading github.com/DataDog/zstd v1.5.5 + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/sync_diff_inspector ./sync_diff_inspector go: downloading github.com/olekukonko/tablewriter v0.0.5 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20241224111350-ad2e4f976c7c go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading gorm.io/driver/mysql v1.5.7 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading modernc.org/sqlite v1.27.0 go: downloading modernc.org/libc v1.37.1 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading github.com/rs/cors v1.7.0 go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:31:44" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.4G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:37 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:37 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:37 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:32 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:32 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:31 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:32 oauth2-server -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:32 sync_diff_inspector + ./bin/cdc version Release Version: v9.0.0-beta.1-10-ge8bfbe4ba Git Commit Hash: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 Git Branch: HEAD UTC Build Time: 2025-04-04 07:31:44 Go Version: go1.23.4 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-4c631d5-e8bfbe4) 1485424640 bytes in 24.27 secs (61191600 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-region-bench ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/pd-ut ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ../third_party_download/bin/xprog ./bin/ + ls -alh ./bin total 3.3G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:38 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:37 bin -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:37 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:37 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:32 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:32 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:31 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:37 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 4 15:37 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:37 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:37 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:37 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:37 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Apr 4 15:37 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 4 15:37 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Apr 4 15:37 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:37 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 4 15:37 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 4 15:37 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 4 15:37 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 4 15:37 minio -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:32 oauth2-server -rwxr-xr-x 1 jenkins jenkins 41M Apr 4 15:37 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:37 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 4 15:37 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:37 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:37 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 4 15:38 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:38 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 4 15:38 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:38 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:38 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:38 stores-dump -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:32 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 241M Apr 4 15:38 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 4 15:38 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 4 15:38 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 4 15:38 xprog [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 78.47 secs (51365457 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-jl8gz-qgd1l --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "07ec694969ee1fbb1651d3caca61dc7bb61af7e5" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-jl8gz" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1ed6922c; decorates RemoteLauncher[hudson.remoting.Channel@3ac8a184:JNLP4-connect connection from 10.233.75.148/10.233.75.148:37248] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] node Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-wzcx3-5m5zr --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "180e6b2227cdfd0b03ba624dc65cb5abb782489a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-wzcx3" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-bd9rv-7tzst --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9691003d2701701e398d9d20e95e527b48bef461" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-bd9rv" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5cc0840b; decorates RemoteLauncher[hudson.remoting.Channel@5a0922f0:JNLP4-connect connection from 10.233.78.65/10.233.78.65:35738] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7ce6838b; decorates RemoteLauncher[hudson.remoting.Channel@25e479ae:JNLP4-connect connection from 10.233.71.9/10.233.71.9:50288] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-nb3mm-ggd1l --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ab92d303a4ad7eea3db70120819562564c2bb3e1" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-nb3mm" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate The recommended git tool is: git [Pipeline] { Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] container Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 [Pipeline] { Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-j3m2d-1fcw6 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8878412d625407e5534e9646e048c33d65fa4751" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-j3m2d" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] checkout The recommended git tool is: git [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@12689108; decorates RemoteLauncher[hudson.remoting.Channel@3da31fb5:JNLP4-connect connection from 10.233.77.227/10.233.77.227:33242] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-p0qwn-n691z --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "93d49925fb8bf7639c878fddf6cb243ec4b377c2" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-p0qwn" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@22ca218b; decorates RemoteLauncher[hudson.remoting.Channel@7a0eac93:JNLP4-connect connection from 10.233.75.71/10.233.75.71:48370] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-z2ljc-x4hsj --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8af731d94d49470c4c555e5a8cd9ae461f6b3cd3" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-z2ljc" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-0s9jj-0fms2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e6d56945949411d966d93c114680000115532da1" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-0s9jj" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-965qc-w0544 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "51cf83a6331d11860614867d6b2ffec66c6e16b0" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-965qc" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-33hs8-pjl6d --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "79318b2c24949e8ec04b1ec376f9846dff63a0d3" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-33hs8" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.22 secs (490327177 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G00 [Pipeline] node [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] container [Pipeline] { The recommended git tool is: git [Pipeline] container [Pipeline] { Run cases: down_db_require_secure_transport bdr_mode capture_suicide_while_balance_table syncpoint syncpoint_check_ts server_config_compatibility changefeed_dup_error_restart safe_mode kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update csv_storage_update_pk_clustered csv_storage_update_pk_nonclustered GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-jl8gz HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=50236c0e-8d61-47ac-a57c-24d6ce2d1476 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G00 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-jl8gz pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/down_db_require_secure_transport/run.sh using Sink-Type: storage... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:01 CST 2025] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz [Pipeline] withCredentials Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@87582b0; decorates RemoteLauncher[hudson.remoting.Channel@11efc8c4:JNLP4-connect connection from 10.233.77.156/10.233.77.156:35720] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@29945eaa; decorates RemoteLauncher[hudson.remoting.Channel@434fa690:JNLP4-connect connection from 10.233.77.208/10.233.77.208:39364] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4af55841; decorates RemoteLauncher[hudson.remoting.Channel@6549def1:JNLP4-connect connection from 10.233.69.182/10.233.69.182:41252] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5da5a838; decorates RemoteLauncher[hudson.remoting.Channel@6beb2a5f:JNLP4-connect connection from 10.233.71.177/10.233.71.177:34780] will be ignored (a typical symptom is the Git executable not being run inside a designated container) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@31cf108a; decorates RemoteLauncher[hudson.remoting.Channel@c832fe0:JNLP4-connect connection from 10.233.71.254/10.233.71.254:52234] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-5q5v3-7hvlq Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3cee5b0bc1dff0fe72df29192d9df40afa820a7f" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-5q5v3" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-55kv6-qljxb --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2dc9464b49e0d51a538209c555701c5402e5d8ff" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-55kv6" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-hfdj1-4vt1h --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "016e7e5fd387caff647cb02e16d82f47334fc31e" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-hfdj1" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-tvh3r-qxrwx --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8616e7a2a50990f8b8038d3338754eec7bf68a8a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-tvh3r" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-lrnft-bgmgh --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "931a884ef5b2f12e850f8772874720ab7f1129e4" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-lrnft" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-mrmkt-hj2b5 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "dabef454c3bcf5cc71e54337accf9b60ccd9b14d" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-mrmkt" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:05 CST 2025] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-42jcc-hwq0q --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "98ad324c19dc606f36def8a7bbd0eb872790410e" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-42jcc" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m is provisioned from template pingcap_tiflow_pull_cdc_integration_storage_test_3706-5zfwb-b6h8d --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "af5047c5ff897209a5982204bc5abb97893eed69" jenkins/label: "pingcap_tiflow_pull_cdc_integration_storage_test_3706-5zfwb" name: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test Running on pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: storage... <<================= kafka downstream isn't support syncpoint record [Fri Apr 4 15:40:10 CST 2025] <<<<<< run test case syncpoint success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.31 secs (484918040 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/syncpoint_check_ts/run.sh using Sink-Type: storage... <<================= kafka downstream isn't support syncpoint record [Fri Apr 4 15:40:14 CST 2025] <<<<<< run test case syncpoint_check_ts success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:19 CST 2025] <<<<<< run test case server_config_compatibility success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.17 secs (493332867 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:24 CST 2025] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/safe_mode/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:28 CST 2025] <<<<<< run test case safe_mode success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.09 secs (498350984 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:33 CST 2025] <<<<<< run test case kafka_big_messages success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:37 CST 2025] <<<<<< run test case kafka_compression success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:42 CST 2025] <<<<<< run test case kafka_messages success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 7.40 secs (544639759 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G02 [Pipeline] sh Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-wzcx3 HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=36a9122d-3b74-4b2d-b667-177ea15cee37 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G02 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-wzcx3 pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:43 CST 2025] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G01 [Pipeline] sh Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth debezium_basic kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-bd9rv HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=7568c527-be04-4a78-82c3-e92a9d9751cd GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G01 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-bd9rv pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:44 CST 2025] <<<<<< run test case http_api success! >>>>>> + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-nb3mm HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=645c8a2e-0746-459d-97ef-deb1d0cbd8a5 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G04 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-nb3mm pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G03 [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { Run cases: row_format drop_many_tables processor_stop_delay partition_table ddl_with_exists GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-j3m2d HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=e04b757a-ce54-4ac0-8f2d-160c4965e678 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G03 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-j3m2d pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@715f54e6; decorates RemoteLauncher[hudson.remoting.Channel@3306f5de:JNLP4-connect connection from 10.233.74.180/10.233.74.180:40978] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@218e51ad; decorates RemoteLauncher[hudson.remoting.Channel@191dbe93:JNLP4-connect connection from 10.233.97.219/10.233.97.219:55604] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@349bb6df; decorates RemoteLauncher[hudson.remoting.Channel@6f62fef2:JNLP4-connect connection from 10.233.93.15/10.233.93.15:35488] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@435d59bc; decorates RemoteLauncher[hudson.remoting.Channel@47227a05:JNLP4-connect connection from 10.233.90.19/10.233.90.19:38040] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@554ee28d; decorates RemoteLauncher[hudson.remoting.Channel@3d52e628:JNLP4-connect connection from 10.233.88.237/10.233.88.237:60496] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4c4d6bd9; decorates RemoteLauncher[hudson.remoting.Channel@548f347:JNLP4-connect connection from 10.233.86.171/10.233.86.171:33976] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@67d23662; decorates RemoteLauncher[hudson.remoting.Channel@5acf796d:JNLP4-connect connection from 10.233.73.81/10.233.73.81:49432] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@248c1608; decorates RemoteLauncher[hudson.remoting.Channel@50f73d45:JNLP4-connect connection from 10.233.86.46/10.233.86.46:33842] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:47 CST 2025] <<<<<< run test case http_api_tls success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:46 CST 2025] <<<<<< run test case kafka_sink_error_resume success! >>>>>> Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: storage... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330D9D5EFF79E7 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:40:50 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 4 15:40:50 CST 2025] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT Verifying downstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:51 CST 2025] <<<<<< run test case mq_sink_lost_callback success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: storage... <<================= Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:54 CST 2025] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:54 CST 2025] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:55 CST 2025] <<<<<< run test case mq_sink_dispatcher success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.15 secs (494476422 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746123100180 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbb1fc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:1353, start at 2025-04-04 15:40:57.361658472 +0800 CST m=+2.301149412 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.343 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.343 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746123100180 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbb1fc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:1353, start at 2025-04-04 15:40:57.361658472 +0800 CST m=+2.301149412 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.343 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.343 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359753090983990 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbb1e40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:1419, start at 2025-04-04 15:40:57.339881116 +0800 CST m=+2.187981723 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.348 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.337 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.337 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:40:59 CST 2025] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746831267929 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbbb14000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:1291, start at 2025-04-04 15:40:57.934975319 +0800 CST m=+2.465019733 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.944 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.951 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.951 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746831267929 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbbb14000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:1291, start at 2025-04-04 15:40:57.934975319 +0800 CST m=+2.465019733 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.944 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.951 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.951 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359754357882203 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbbca40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:1364, start at 2025-04-04 15:40:58.028261898 +0800 CST m=+2.472273080 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:58.037 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:58.025 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:58.025 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting Upstream PD... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:00 CST 2025] <<<<<< run test case kafka_column_selector success! >>>>>> Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2659.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2680.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114245357109249 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114245357109249 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:03 CST 2025] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.27002702.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114245490802689 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114245490802689 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:03 CST 2025] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.27202722.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:03 CST 2025] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:04 CST 2025] <<<<<< run test case kafka_column_selector_avro success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:06 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/35cb29be-1212-4c48-963a-90cd45e2cc96 {"id":"35cb29be-1212-4c48-963a-90cd45e2cc96","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecb75e9 35cb29be-1212-4c48-963a-90cd45e2cc96 /tidb/cdc/default/default/upstream/7489359746123100180 {"id":7489359746123100180,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/35cb29be-1212-4c48-963a-90cd45e2cc96 {"id":"35cb29be-1212-4c48-963a-90cd45e2cc96","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecb75e9 35cb29be-1212-4c48-963a-90cd45e2cc96 /tidb/cdc/default/default/upstream/7489359746123100180 {"id":7489359746123100180,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/35cb29be-1212-4c48-963a-90cd45e2cc96 {"id":"35cb29be-1212-4c48-963a-90cd45e2cc96","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecb75e9 35cb29be-1212-4c48-963a-90cd45e2cc96 /tidb/cdc/default/default/upstream/7489359746123100180 {"id":7489359746123100180,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2759.out cli changefeed create --start-ts=457114245357109249 '--sink-uri=file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-17736?protocol=canal-json&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/foreign_key/conf/cf.toml Create changefeed successfully! ID: 49cfe976-b89a-4aa1-8e7d-b19440af770d Info: {"upstream_id":7489359746123100180,"namespace":"default","id":"49cfe976-b89a-4aa1-8e7d-b19440af770d","sink_uri":"file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-17736?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:06.767005174+08:00","start_ts":457114245357109249,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!foreign_key.t12"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114245357109249,"checkpoint_ts":457114245357109249,"checkpoint_time":"2025-04-04 15:41:01.842"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:06 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 {"id":"f8d2cabb-0d31-48e9-a64f-d0f27d9190e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752464} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecd0de9 f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 /tidb/cdc/default/default/upstream/7489359746831267929 {"id":7489359746831267929,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 {"id":"f8d2cabb-0d31-48e9-a64f-d0f27d9190e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752464} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecd0de9 f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 /tidb/cdc/default/default/upstream/7489359746831267929 {"id":7489359746831267929,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 {"id":"f8d2cabb-0d31-48e9-a64f-d0f27d9190e6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752464} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecd0de9 f8d2cabb-0d31-48e9-a64f-d0f27d9190e6 /tidb/cdc/default/default/upstream/7489359746831267929 {"id":7489359746831267929,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2776.out cli changefeed create --start-ts=457114245490802689 '--sink-uri=file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9029?protocol=canal-json&enable-tidb-extension=true' Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Create changefeed successfully! ID: 4e0c42c3-8768-4d8d-a875-2f5d9abd0a1f Info: {"upstream_id":7489359746831267929,"namespace":"default","id":"4e0c42c3-8768-4d8d-a875-2f5d9abd0a1f","sink_uri":"file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9029?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:07.332998844+08:00","start_ts":457114245490802689,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114245490802689,"checkpoint_ts":457114245490802689,"checkpoint_time":"2025-04-04 15:41:02.352"} PASS Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.5% of statements in github.com/pingcap/tiflow/... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 9.28 secs (434173205 bytes/sec) [Pipeline] { [Pipeline] cache + set +x + workdir=/tmp/tidb_cdc_test/foreign_key + sink_uri='file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-17736?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/foreign_key ++ date + echo '[Fri Apr 4 15:41:08 CST 2025] <<<<<< START storage consumer in foreign_key case >>>>>>' [Fri Apr 4 15:41:08 CST 2025] <<<<<< START storage consumer in foreign_key case >>>>>> + cd /tmp/tidb_cdc_test/foreign_key + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/foreign_key + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/foreign_key/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-17736?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + workdir=/tmp/tidb_cdc_test/row_format + sink_uri='file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9029?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/row_format ++ date + echo '[Fri Apr 4 15:41:08 CST 2025] <<<<<< START storage consumer in row_format case >>>>>>' [Fri Apr 4 15:41:08 CST 2025] <<<<<< START storage consumer in row_format case >>>>>> + cd /tmp/tidb_cdc_test/row_format + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/row_format + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/row_format/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9029?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table foreign_key.finish_mark not exists for 1-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: storage... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: storage... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DA20D815685 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:41:10 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 4 15:41:10 CST 2025] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT table foreign_key.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359808356130824 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc73f0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:1777, start at 2025-04-04 15:41:09.780736973 +0800 CST m=+2.575863070 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:09.788 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:09.756 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:09.756 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359808356130824 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc73f0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:1777, start at 2025-04-04 15:41:09.780736973 +0800 CST m=+2.575863070 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:09.788 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:09.756 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:09.756 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359797991046434 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc73fc0001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:1850, start at 2025-04-04 15:41:09.759982664 +0800 CST m=+2.471986104 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:09.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:09.759 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:09.759 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table foreign_key.finish_mark not exists for 3-th check, retry later The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 start tidb cluster in /tmp/tidb_cdc_test/lossy_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table foreign_key.finish_mark not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:15 CST 2025] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table foreign_key.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 7.70 secs (523636490 bytes/sec) [Pipeline] { [Pipeline] cache Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 1-th check, retry later table foreign_key.finish_mark not exists for 6-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:19 CST 2025] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 7-th check, retry later table row_format.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359860910571152 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd28b0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:3375, start at 2025-04-04 15:41:21.364844089 +0800 CST m=+1.539973218 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:21.372 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:21.374 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:21.374 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + pd_host=127.0.0.1 + pd_port=2579 + is_tls=true + '[' true == true ']' ++ run_cdc_cli tso query --pd=https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3416.out cli tso query --pd=https://127.0.0.1:2579 table foreign_key.finish_mark not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359854794529248 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd23cc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:2643, start at 2025-04-04 15:41:21.044581356 +0800 CST m=+2.491076406 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:21.053 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:21.011 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:21.011 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359854794529248 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd23cc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:2643, start at 2025-04-04 15:41:21.044581356 +0800 CST m=+2.491076406 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:21.053 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:21.011 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:21.011 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359855042537764 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd2404000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:2715, start at 2025-04-04 15:41:21.064449951 +0800 CST m=+2.409605327 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:21.073 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:21.066 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:21.066 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table row_format.finish_mark not exists for 4-th check, retry later + set +x + tso='457114250726342659 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114250726342659 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:23 CST 2025] <<<<<< START cdc server in cli_tls_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.34563458.out server --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_cli_tls_with_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_data_cli_tls_with_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/cli_tls_with_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:24 CST 2025] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> [Fri Apr 4 15:41:25 CST 2025] <<<<<< START cdc server in lossy_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.lossy_ddl.39433945.out server --log-file /tmp/tidb_cdc_test/lossy_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/lossy_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table foreign_key.finish_mark not exists for 9-th check, retry later table row_format.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:26 GMT < Content-Length: 1242 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd00202e-8a28-4411-be7e-b887595a54af {"id":"cd00202e-8a28-4411-be7e-b887595a54af","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf2c6ef1 cd00202e-8a28-4411-be7e-b887595a54af /tidb/cdc/default/default/upstream/7489359860910571152 {"id":7489359860910571152,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd00202e-8a28-4411-be7e-b887595a54af {"id":"cd00202e-8a28-4411-be7e-b887595a54af","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf2c6ef1 cd00202e-8a28-4411-be7e-b887595a54af /tidb/cdc/default/default/upstream/7489359860910571152 {"id":7489359860910571152,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cd00202e-8a28-4411-be7e-b887595a54af {"id":"cd00202e-8a28-4411-be7e-b887595a54af","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf2c6ef1 cd00202e-8a28-4411-be7e-b887595a54af /tidb/cdc/default/default/upstream/7489359860910571152 {"id":7489359860910571152,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x table foreign_key.finish_mark not exists for 10-th check, retry later table row_format.finish_mark not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:28 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b {"id":"f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752485} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf2d31d8 f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b /tidb/cdc/default/default/upstream/7489359854794529248 {"id":7489359854794529248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b {"id":"f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752485} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf2d31d8 f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b /tidb/cdc/default/default/upstream/7489359854794529248 {"id":7489359854794529248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b {"id":"f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752485} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf2d31d8 f67f5571-5c1e-42a5-b42a-d0bf16ac0d3b /tidb/cdc/default/default/upstream/7489359854794529248 {"id":7489359854794529248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: b55377de-99d5-44b8-9f36-54f610442a84 Info: {"upstream_id":7489359854794529248,"namespace":"default","id":"b55377de-99d5-44b8-9f36-54f610442a84","sink_uri":"blackhole:","create_time":"2025-04-04T15:41:28.275337435+08:00","start_ts":457114252256477189,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114252256477189,"checkpoint_ts":457114252256477189,"checkpoint_time":"2025-04-04 15:41:28.161"} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3523.out cli changefeed create --start-ts=457114250726342659 '--sink-uri=file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name table foreign_key.finish_mark not exists for 11-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:41:28 CST 2025] <<<<<< run test case multi_topics_v2 success! >>>>>> [WARN] --tz is deprecated in changefeed settings. Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 9.28 secs (434158764 bytes/sec) [Pipeline] { Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359860910571152,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:29.480192987+08:00","start_ts":457114250726342659,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114250726342659,"checkpoint_ts":457114250726342659,"checkpoint_time":"2025-04-04 15:41:22.324"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] cache table row_format.finish_mark not exists for 7-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/cli_tls_with_auth + sink_uri='file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/cli_tls_with_auth ++ date + echo '[Fri Apr 4 15:41:30 CST 2025] <<<<<< START storage consumer in cli_tls_with_auth case >>>>>>' [Fri Apr 4 15:41:30 CST 2025] <<<<<< START storage consumer in cli_tls_with_auth case >>>>>> + cd /tmp/tidb_cdc_test/cli_tls_with_auth + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/cli_tls_with_auth + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 1-th check, retry later table foreign_key.finish_mark not exists for 12-th check, retry later table row_format.finish_mark not exists for 8-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 1-th time, retry later table foreign_key.finish_mark not exists for 13-th check, retry later table test.simple not exists for 2-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=storage + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run storage + '[' storage '!=' storage ']' + rm -rf /tmp/tidb_cdc_test/storage_cleanup + mkdir -p /tmp/tidb_cdc_test/storage_cleanup + start_tidb_cluster --workdir /tmp/tidb_cdc_test/storage_cleanup The 1 times to try to start tidb cluster... check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 2-th time, retry later table row_format.finish_mark not exists for 9-th check, retry later table foreign_key.finish_mark not exists for 14-th check, retry later table test.simple not exists for 3-th check, retry later table row_format.finish_mark not exists for 10-th check, retry later table foreign_key.finish_mark not exists for 15-th check, retry later table test.simple not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/storage_cleanup Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table row_format.finish_mark not exists for 11-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 3-th time, retry later table foreign_key.finish_mark not exists for 16-th check, retry later table test.simple not exists for 5-th check, retry later table row_format.finish_mark not exists for 12-th check, retry later Verifying downstream PD is started... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.61 secs (467869948 bytes/sec) [Pipeline] { [Pipeline] sh table foreign_key.finish_mark not exists for 17-th check, retry later table test.simple not exists for 6-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G05 [Pipeline] sh Run cases: charset_gbk ddl_manager multi_source vector GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-p0qwn HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=e187eb89-b7ef-49c0-a51a-a1885996b256 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G05 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0 pingcap_tiflow_pull_cdc_integration_storage_test_3706-p0qwn PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G07 [Pipeline] sh Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Run cases: kv_client_stream_reconnect cdc split_region GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-z2ljc HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=3ecf246a-5b2e-4c83-9ea9-719a8e99b539 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G07 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-z2ljc pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G06 [Pipeline] sh Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-0s9jj HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=3315c849-7ecb-4ca2-ac35-846d2ac32451 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G06 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr pingcap_tiflow_pull_cdc_integration_storage_test_3706-0s9jj PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table row_format.finish_mark not exists for 13-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-965qc HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=9fee6c6a-cb60-4b0b-8c1d-a4302b4682ce GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G08 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58 pingcap_tiflow_pull_cdc_integration_storage_test_3706-965qc PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G09 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-33hs8 HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=7503b34a-21fd-4157-b983-7da5261a9662 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G09 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-33hs8 pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table test.simple exists table test.`simple-dash` exists + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114256113401861, "checkpoint_time": "2025-04-04 15:41:42.874", "error": null }' + echo '{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114256113401861, "checkpoint_time": "2025-04-04 15:41:42.874", "error": null }' { "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114256113401861, "checkpoint_time": "2025-04-04 15:41:42.874", "error": null } ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114256113401861, '"checkpoint_time":' '"2025-04-04' '15:41:42.874",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114256113401861, '"checkpoint_time":' '"2025-04-04' '15:41:42.874",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: https://127.0.0.1:2579 table foreign_key.finish_mark not exists for 18-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 14-th check, retry later Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0038b9120} {Type:update Path:[Config SyncPointRetention] From: To:0xc0038b9128} {Type:update Path:[Config Consistent] From: To:0xc0009cc850} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3762.out cli changefeed --changefeed-id custom-changefeed-name pause check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 4-th time, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv table foreign_key.finish_mark not exists for 19-th check, retry later [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { table row_format.finish_mark not exists for 15-th check, retry later [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { + set +x start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { table foreign_key.finish_mark not exists for 20-th check, retry later [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow table row_format.finish_mark not exists for 16-th check, retry later [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359957893190733 TiDB Cluster ID. tikv_gc_leader_uuid 657fefea8b40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:2376, start at 2025-04-04 15:41:45.924927803 +0800 CST m=+2.311648784 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:45.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:45.901 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:45.901 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359957893190733 TiDB Cluster ID. tikv_gc_leader_uuid 657fefea8b40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:2376, start at 2025-04-04 15:41:45.924927803 +0800 CST m=+2.311648784 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:45.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:45.901 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:45.901 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359961106085143 TiDB Cluster ID. tikv_gc_leader_uuid 657fefea978000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:2443, start at 2025-04-04 15:41:45.96777544 +0800 CST m=+2.281469142 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:45.975 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:45.950 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:45.950 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table foreign_key.finish_mark exists check diff successfully table foreign_key.t12 does not exists + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114256375545863, "checkpoint_time": "2025-04-04 15:41:43.874", "error": null }' + echo '{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114256375545863, "checkpoint_time": "2025-04-04 15:41:43.874", "error": null }' { "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114256375545863, "checkpoint_time": "2025-04-04 15:41:43.874", "error": null } ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114256375545863, '"checkpoint_time":' '"2025-04-04' '15:41:43.874",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114256375545863, '"checkpoint_time":' '"2025-04-04' '15:41:43.874",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3835.out cli changefeed update --pd=https://127.0.0.1:2579 --config=/tmp/tidb_cdc_test/cli_tls_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name wait process cdc.test exit for 1-th time... table row_format.finish_mark not exists for 17-th check, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc003a75ab8} {Type:update Path:[Config SyncPointRetention] From: To:0xc003a75ac8} {Type:update Path:[Config Consistent] From: To:0xc000e99ce0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359860910571152,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:29.480192987+08:00","start_ts":457114250726342659,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114256375545863,"checkpoint_time":"2025-04-04 15:41:43.874"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + cd /tmp/tidb_cdc_test/storage_cleanup + run_cdc_server --workdir /tmp/tidb_cdc_test/storage_cleanup --binary cdc.test [Fri Apr 4 15:41:50 CST 2025] <<<<<< START cdc server in storage_cleanup case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.37003702.out server --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_cleanup/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:50 CST 2025] <<<<<< run test case foreign_key success! >>>>>> + set +x Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3867.out cli changefeed --changefeed-id custom-changefeed-name resume Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 18-th check, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:53 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de07e26-8020-48e8-ba94-f9f0bb487c8a {"id":"0de07e26-8020-48e8-ba94-f9f0bb487c8a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752510} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8923d8 0de07e26-8020-48e8-ba94-f9f0bb487c8a /tidb/cdc/default/default/upstream/7489359957893190733 {"id":7489359957893190733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de07e26-8020-48e8-ba94-f9f0bb487c8a {"id":"0de07e26-8020-48e8-ba94-f9f0bb487c8a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752510} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8923d8 0de07e26-8020-48e8-ba94-f9f0bb487c8a /tidb/cdc/default/default/upstream/7489359957893190733 {"id":7489359957893190733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de07e26-8020-48e8-ba94-f9f0bb487c8a {"id":"0de07e26-8020-48e8-ba94-f9f0bb487c8a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752510} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8923d8 0de07e26-8020-48e8-ba94-f9f0bb487c8a /tidb/cdc/default/default/upstream/7489359957893190733 {"id":7489359957893190733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + generate_historic_files + local target_bucket=storage_test ++ date -d yesterday +%Y-%m-%d + yesterday=2025-04-03 ++ date -d '2 days ago' +%Y-%m-%d + day_before_yesterday=2025-04-02 + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type 2025-04-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type + local day=2025-04-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type_dummy 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type_dummy + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset_dummy 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset_dummy + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns_dummy 2025-04-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns_dummy + local day=2025-04-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/8.data Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_data_type 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_data_type + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_charset 2025-04-02 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_charset + local day=2025-04-02 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 binary_columns 2025-04-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=binary_columns + local day=2025-04-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_data_type 2022-01-01 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_data_type + local day=2022-01-01 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_charset 2022-01-02 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_charset + local day=2022-01-02 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test binary_columns 2022-01-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=binary_columns + local day=2022-01-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 ++ seq 1 10 + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in $(seq 1 $file_cnt) + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta/CDC.index + SINK_URI_DEFAULT='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.3934.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: default-config-test Info: {"upstream_id":7489359957893190733,"namespace":"default","id":"default-config-test","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s","create_time":"2025-04-04T15:41:53.986173317+08:00","start_ts":457114259004063750,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114259004063750,"checkpoint_ts":457114259004063750,"checkpoint_time":"2025-04-04 15:41:53.901"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 19-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359989062470665 TiDB Cluster ID. tikv_gc_leader_uuid 657feff18a4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:1360, start at 2025-04-04 15:41:53.095099789 +0800 CST m=+2.408761024 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:53.103 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:53.065 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:53.065 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359989062470665 TiDB Cluster ID. tikv_gc_leader_uuid 657feff18a4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:1360, start at 2025-04-04 15:41:53.095099789 +0800 CST m=+2.408761024 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:53.103 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:53.065 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:53.065 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359992234204107 TiDB Cluster ID. tikv_gc_leader_uuid 657feff1a40000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:1416, start at 2025-04-04 15:41:53.201512787 +0800 CST m=+2.421929691 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:53.210 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:53.218 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:53.218 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + sleep 20 table row_format.finish_mark not exists for 20-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359996438195624 TiDB Cluster ID. tikv_gc_leader_uuid 657feff2c080006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:1352, start at 2025-04-04 15:41:54.341466817 +0800 CST m=+2.412100017 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.355 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.355 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359996438195624 TiDB Cluster ID. tikv_gc_leader_uuid 657feff2c080006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:1352, start at 2025-04-04 15:41:54.341466817 +0800 CST m=+2.412100017 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.355 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.355 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359989920933176 TiDB Cluster ID. tikv_gc_leader_uuid 657feff2cc4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:1426, start at 2025-04-04 15:41:54.398303543 +0800 CST m=+2.356183867 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.436 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.403 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.403 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359990447564738 TiDB Cluster ID. tikv_gc_leader_uuid 657feff32500008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:1354, start at 2025-04-04 15:41:54.720848113 +0800 CST m=+2.694819115 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.759 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.759 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359990447564738 TiDB Cluster ID. tikv_gc_leader_uuid 657feff32500008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:1354, start at 2025-04-04 15:41:54.720848113 +0800 CST m=+2.694819115 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.759 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.759 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359994932384808 TiDB Cluster ID. tikv_gc_leader_uuid 657feff36300007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:1419, start at 2025-04-04 15:41:54.973271048 +0800 CST m=+2.874088789 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:54.983 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:54.956 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:54.956 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359991224451259 TiDB Cluster ID. tikv_gc_leader_uuid 657feff3970000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:1280, start at 2025-04-04 15:41:55.183365804 +0800 CST m=+2.901055904 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.191 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.164 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.164 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359991224451259 TiDB Cluster ID. tikv_gc_leader_uuid 657feff3970000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:1280, start at 2025-04-04 15:41:55.183365804 +0800 CST m=+2.901055904 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.191 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.164 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.164 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360000765975332 TiDB Cluster ID. tikv_gc_leader_uuid 657feff3b04000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:1350, start at 2025-04-04 15:41:55.29271022 +0800 CST m=+2.909274684 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.302 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114259534643202, "checkpoint_time": "2025-04-04 15:41:55.925", "error": null }' + echo '{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114259534643202, "checkpoint_time": "2025-04-04 15:41:55.925", "error": null }' { "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114259534643202, "checkpoint_time": "2025-04-04 15:41:55.925", "error": null } ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114259534643202, '"checkpoint_time":' '"2025-04-04' '15:41:55.925",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114259534643202, '"checkpoint_time":' '"2025-04-04' '15:41:55.925",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3930.out cli changefeed --changefeed-id custom-changefeed-name remove [Fri Apr 4 15:41:57 CST 2025] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.26382640.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360004575893062 TiDB Cluster ID. tikv_gc_leader_uuid 657feff3de80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:1362, start at 2025-04-04 15:41:55.463598063 +0800 CST m=+2.275850916 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.471 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.450 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.450 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360004575893062 TiDB Cluster ID. tikv_gc_leader_uuid 657feff3de80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:1362, start at 2025-04-04 15:41:55.463598063 +0800 CST m=+2.275850916 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.471 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.450 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.450 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360006288833868 TiDB Cluster ID. tikv_gc_leader_uuid 657feff40b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:1425, start at 2025-04-04 15:41:55.648636508 +0800 CST m=+2.387187809 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:55.655 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:55.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:55.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 7.44 secs (541887299 bytes/sec) [Pipeline] { [Pipeline] cache Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114259796787202 SinkURI: file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 21-th check, retry later [Fri Apr 4 15:41:58 CST 2025] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.26672669.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:41:58 CST 2025] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26862688.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2667.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 4 15:41:59 CST 2025] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27302732.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table row_format.finish_mark not exists for 22-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:00 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c41139a8-d7cf-4037-a42b-1c7aa649fc28 {"id":"c41139a8-d7cf-4037-a42b-1c7aa649fc28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752517} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfa8a4d9 c41139a8-d7cf-4037-a42b-1c7aa649fc28 /tidb/cdc/default/default/upstream/7489359989062470665 {"id":7489359989062470665,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c41139a8-d7cf-4037-a42b-1c7aa649fc28 {"id":"c41139a8-d7cf-4037-a42b-1c7aa649fc28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752517} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfa8a4d9 c41139a8-d7cf-4037-a42b-1c7aa649fc28 /tidb/cdc/default/default/upstream/7489359989062470665 {"id":7489359989062470665,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c41139a8-d7cf-4037-a42b-1c7aa649fc28 {"id":"c41139a8-d7cf-4037-a42b-1c7aa649fc28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752517} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfa8a4d9 c41139a8-d7cf-4037-a42b-1c7aa649fc28 /tidb/cdc/default/default/upstream/7489359989062470665 {"id":7489359989062470665,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/kv_client_stream_reconnect + sink_uri='file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-28731?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/kv_client_stream_reconnect ++ date + echo '[Fri Apr 4 15:42:00 CST 2025] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>>' [Fri Apr 4 15:42:00 CST 2025] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>> + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-28731?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + tso='457114260589772801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114260589772801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:01 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/226e2574-1e97-4edc-962b-da0cf53b2b90 {"id":"226e2574-1e97-4edc-962b-da0cf53b2b90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabbad7 226e2574-1e97-4edc-962b-da0cf53b2b90 /tidb/cdc/default/default/upstream/7489359996438195624 {"id":7489359996438195624,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/226e2574-1e97-4edc-962b-da0cf53b2b90 {"id":"226e2574-1e97-4edc-962b-da0cf53b2b90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabbad7 226e2574-1e97-4edc-962b-da0cf53b2b90 /tidb/cdc/default/default/upstream/7489359996438195624 {"id":7489359996438195624,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/226e2574-1e97-4edc-962b-da0cf53b2b90 {"id":"226e2574-1e97-4edc-962b-da0cf53b2b90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabbad7 226e2574-1e97-4edc-962b-da0cf53b2b90 /tidb/cdc/default/default/upstream/7489359996438195624 {"id":7489359996438195624,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 2702166f-59d7-407e-8f65-7619c76a63ff Info: {"upstream_id":7489359996438195624,"namespace":"default","id":"2702166f-59d7-407e-8f65-7619c76a63ff","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:01.734305231+08:00","start_ts":457114260197867523,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114260197867523,"checkpoint_ts":457114260197867523,"checkpoint_time":"2025-04-04 15:41:58.455"} + workdir=/tmp/tidb_cdc_test/charset_gbk + sink_uri=mysql://normal:123456@127.0.0.1:3306/ + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/charset_gbk ++ date + echo '[Fri Apr 4 15:42:01 CST 2025] <<<<<< START storage consumer in charset_gbk case >>>>>>' [Fri Apr 4 15:42:01 CST 2025] <<<<<< START storage consumer in charset_gbk case >>>>>> + cd /tmp/tidb_cdc_test/charset_gbk + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/charset_gbk + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/charset_gbk/cdc_storage_consumer.log --log-level debug --upstream-uri mysql://normal:123456@127.0.0.1:3306/ --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:01 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee {"id":"0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabefd8 0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee /tidb/cdc/default/default/upstream/7489359990447564738 {"id":7489359990447564738,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee {"id":"0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabefd8 0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee /tidb/cdc/default/default/upstream/7489359990447564738 {"id":7489359990447564738,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee {"id":"0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752518} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfabefd8 0aa0e0e3-399e-4783-81da-9f2b6cd5b5ee /tidb/cdc/default/default/upstream/7489359990447564738 {"id":7489359990447564738,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/gc_safepoint + sink_uri='file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-10992?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/gc_safepoint ++ date + echo '[Fri Apr 4 15:42:01 CST 2025] <<<<<< START storage consumer in gc_safepoint case >>>>>>' [Fri Apr 4 15:42:01 CST 2025] <<<<<< START storage consumer in gc_safepoint case >>>>>> + cd /tmp/tidb_cdc_test/gc_safepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/gc_safepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-10992?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' 0 table row_format.finish_mark not exists for 23-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 6-th time, retry later changefeed count 0 check pass, pd_addr: https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3996.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name ***************** properties ***************** "updateproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "scanproportion"="0" "threadcount"="2" "recordcount"="10" "mysql.db"="sink_retry" "dotransactions"="false" "mysql.port"="4000" "operationcount"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "workload"="core" "insertproportion"="0" "readproportion"="0" ********************************************** Run finished, takes 10.251698ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1821.6, Avg(us): 1955, Min(us): 1103, Max(us): 4694, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:42:02 CST 2025] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.27332735.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359860910571152,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-30652?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:02.778999921+08:00","start_ts":457114261304115205,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114261304115205,"checkpoint_ts":457114261304115205,"checkpoint_time":"2025-04-04 15:42:02.675"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:02 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/892a3cd2-783e-4fac-bcd2-a05af826c1c9 {"id":"892a3cd2-783e-4fac-bcd2-a05af826c1c9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752520} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaafc38 892a3cd2-783e-4fac-bcd2-a05af826c1c9 /tidb/cdc/default/default/upstream/7489359991224451259 {"id":7489359991224451259,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/892a3cd2-783e-4fac-bcd2-a05af826c1c9 {"id":"892a3cd2-783e-4fac-bcd2-a05af826c1c9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752520} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaafc38 892a3cd2-783e-4fac-bcd2-a05af826c1c9 /tidb/cdc/default/default/upstream/7489359991224451259 {"id":7489359991224451259,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/892a3cd2-783e-4fac-bcd2-a05af826c1c9 {"id":"892a3cd2-783e-4fac-bcd2-a05af826c1c9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752520} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaafc38 892a3cd2-783e-4fac-bcd2-a05af826c1c9 /tidb/cdc/default/default/upstream/7489359991224451259 {"id":7489359991224451259,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_err_chan + sink_uri='file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-24767?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_err_chan ++ date + echo '[Fri Apr 4 15:42:03 CST 2025] <<<<<< START storage consumer in processor_err_chan case >>>>>>' [Fri Apr 4 15:42:03 CST 2025] <<<<<< START storage consumer in processor_err_chan case >>>>>> + cd /tmp/tidb_cdc_test/processor_err_chan + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_err_chan + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-24767?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 ab2d5161-9d48-408c-bfb9-c140738296b4 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=ab2d5161-9d48-408c-bfb9-c140738296b4 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c ab2d5161-9d48-408c-bfb9-c140738296b4 -s + info='{ "upstream_id": 7489359991224451259, "namespace": "default", "id": "ab2d5161-9d48-408c-bfb9-c140738296b4", "state": "normal", "checkpoint_tso": 457114261379874818, "checkpoint_time": "2025-04-04 15:42:02.964", "error": null }' + echo '{ "upstream_id": 7489359991224451259, "namespace": "default", "id": "ab2d5161-9d48-408c-bfb9-c140738296b4", "state": "normal", "checkpoint_tso": 457114261379874818, "checkpoint_time": "2025-04-04 15:42:02.964", "error": null }' { "upstream_id": 7489359991224451259, "namespace": "default", "id": "ab2d5161-9d48-408c-bfb9-c140738296b4", "state": "normal", "checkpoint_tso": 457114261379874818, "checkpoint_time": "2025-04-04 15:42:02.964", "error": null } ++ echo '{' '"upstream_id":' 7489359991224451259, '"namespace":' '"default",' '"id":' '"ab2d5161-9d48-408c-bfb9-c140738296b4",' '"state":' '"normal",' '"checkpoint_tso":' 457114261379874818, '"checkpoint_time":' '"2025-04-04' '15:42:02.964",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359991224451259, '"namespace":' '"default",' '"id":' '"ab2d5161-9d48-408c-bfb9-c140738296b4",' '"state":' '"normal",' '"checkpoint_tso":' 457114261379874818, '"checkpoint_time":' '"2025-04-04' '15:42:02.964",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check diff failed 1-th time, retry later + set +x table row_format.finish_mark not exists for 24-th check, retry later check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:05 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e73ddbde-6f2e-4951-a82b-7f58e8d3e98e {"id":"e73ddbde-6f2e-4951-a82b-7f58e8d3e98e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752522} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaea2fc e73ddbde-6f2e-4951-a82b-7f58e8d3e98e /tidb/cdc/default/default/upstream/7489360004575893062 {"id":7489360004575893062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e73ddbde-6f2e-4951-a82b-7f58e8d3e98e {"id":"e73ddbde-6f2e-4951-a82b-7f58e8d3e98e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752522} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaea2fc e73ddbde-6f2e-4951-a82b-7f58e8d3e98e /tidb/cdc/default/default/upstream/7489360004575893062 {"id":7489360004575893062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e73ddbde-6f2e-4951-a82b-7f58e8d3e98e {"id":"e73ddbde-6f2e-4951-a82b-7f58e8d3e98e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752522} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfaea2fc e73ddbde-6f2e-4951-a82b-7f58e8d3e98e /tidb/cdc/default/default/upstream/7489360004575893062 {"id":7489360004575893062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2788.out cli changefeed create --start-ts=457114260589772801 '--sink-uri=file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-4715?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: ccffd8a9-ff4a-4ba2-b30e-9d6882f35ec7 Info: {"upstream_id":7489360004575893062,"namespace":"default","id":"ccffd8a9-ff4a-4ba2-b30e-9d6882f35ec7","sink_uri":"file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-4715?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:05.971168017+08:00","start_ts":457114260589772801,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114260589772801,"checkpoint_ts":457114260589772801,"checkpoint_time":"2025-04-04 15:41:59.950"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 25-th check, retry later check diff failed 2-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/sink_retry + sink_uri='file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-4715?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_retry ++ date + echo '[Fri Apr 4 15:42:07 CST 2025] <<<<<< START storage consumer in sink_retry case >>>>>>' [Fri Apr 4 15:42:07 CST 2025] <<<<<< START storage consumer in sink_retry case >>>>>> + cd /tmp/tidb_cdc_test/sink_retry + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_retry + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_retry/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-4715?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114262417965059, "checkpoint_time": "2025-04-04 15:42:06.924", "error": null }' + echo '{ "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114262417965059, "checkpoint_time": "2025-04-04 15:42:06.924", "error": null }' { "upstream_id": 7489359860910571152, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114262417965059, "checkpoint_time": "2025-04-04 15:42:06.924", "error": null } ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114262417965059, '"checkpoint_time":' '"2025-04-04' '15:42:06.924",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359860910571152, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114262417965059, '"checkpoint_time":' '"2025-04-04' '15:42:06.924",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] check diff failed 3-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.03 secs (501652205 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 26-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4085.out cli unsafe delete-service-gc-safepoint start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later Confirm that you know what this command will do and use it at your own risk [Y/N] check diff failed 3-th time, retry later CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 4-th time, retry later table charset_gbk_test1.t0 not exists for 2-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4122.out cli unsafe reset --no-confirm --pd=https://127.0.0.1:2579 table row_format.finish_mark not exists for 27-th check, retry later reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 4-th time, retry later check diff failed 5-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later + set +x table row_format.finish_mark exists check diff successfully check diff failed 5-th time, retry later wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... check diff successfully check_safepoint_forward http://127.0.0.1:2379 7489359990447564738 457114263764074504 457114261024669699 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:42:14 CST 2025] <<<<<< run test case row_format success! >>>>>> run task successfully check_changefeed_state http://127.0.0.1:2379 f93678ed-53bf-4b12-85e1-675dd37d6f13 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=f93678ed-53bf-4b12-85e1-675dd37d6f13 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f93678ed-53bf-4b12-85e1-675dd37d6f13 -s check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 7-th time, retry later check diff failed 6-th time, retry later + info='{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' + echo '{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' { "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null } ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"stopped",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"stopped",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359990447564738 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4206.out cli unsafe resolve-lock --region=16 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 3-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 7.06 secs (570822101 bytes/sec) [Pipeline] { [Pipeline] cache + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4231.out cli unsafe resolve-lock --region=16 --ts=457114263715577862 check diff successfully PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + check_file_exists true + local all_should_exist=true + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/10.data ']' + SINK_URI='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.4020.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml Create changefeed successfully! ID: 353244cc-e2f5-4dfe-9bcc-b50d03937254 Info: {"upstream_id":7489359957893190733,"namespace":"default","id":"353244cc-e2f5-4dfe-9bcc-b50d03937254","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s","create_time":"2025-04-04T15:42:15.835997842+08:00","start_ts":457114264731910148,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"file_expiration_days":1,"file_cleanup_cron_spec":"* * * * * *"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114264731910148,"checkpoint_ts":457114264731910148,"checkpoint_time":"2025-04-04 15:42:15.751"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360089931700139 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0079400012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:3883, start at 2025-04-04 15:42:15.651545993 +0800 CST m=+2.340627642 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:15.658 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:15.632 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:15.632 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360089931700139 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0079400012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:3883, start at 2025-04-04 15:42:15.651545993 +0800 CST m=+2.340627642 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:15.658 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:15.632 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:15.632 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360083330517126 TiDB Cluster ID. tikv_gc_leader_uuid 657ff007940000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:3950, start at 2025-04-04 15:42:15.680470826 +0800 CST m=+2.296347518 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:15.732 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:15.732 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:15.732 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark exists check table exists success check diff successfully wait process cdc.test exit for 2-th time... run task successfully check_changefeed_state http://127.0.0.1:2379 f93678ed-53bf-4b12-85e1-675dd37d6f13 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=f93678ed-53bf-4b12-85e1-675dd37d6f13 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f93678ed-53bf-4b12-85e1-675dd37d6f13 -s wait process cdc.test exit for 1-th time... + info='{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "normal", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' + echo '{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "normal", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' { "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "normal", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null } ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"normal",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"normal",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7489359990447564738 457114264288362503 457114264288362504 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:18 CST 2025] <<<<<< run test case processor_err_chan success! >>>>>> + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 11777 333 --:--:-- --:--:-- --:--:-- 12111 { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:19 CST 2025] <<<<<< run test case charset_gbk success! >>>>>> run task successfully check_changefeed_state http://127.0.0.1:2379 f93678ed-53bf-4b12-85e1-675dd37d6f13 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=f93678ed-53bf-4b12-85e1-675dd37d6f13 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f93678ed-53bf-4b12-85e1-675dd37d6f13 -s + info='{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' + echo '{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null }' { "upstream_id": 7489359990447564738, "namespace": "default", "id": "f93678ed-53bf-4b12-85e1-675dd37d6f13", "state": "stopped", "checkpoint_tso": 457114264288362504, "checkpoint_time": "2025-04-04 15:42:14.059", "error": null } ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"stopped",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"f93678ed-53bf-4b12-85e1-675dd37d6f13",' '"state":' '"stopped",' '"checkpoint_tso":' 457114264288362504, '"checkpoint_time":' '"2025-04-04' '15:42:14.059",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 eb2b9e33-5d95-4666-9630-8fb8f8130dae normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=eb2b9e33-5d95-4666-9630-8fb8f8130dae + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c eb2b9e33-5d95-4666-9630-8fb8f8130dae -s + info='{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "eb2b9e33-5d95-4666-9630-8fb8f8130dae", "state": "normal", "checkpoint_tso": 457114265808535554, "checkpoint_time": "2025-04-04 15:42:19.858", "error": null }' + echo '{ "upstream_id": 7489359990447564738, "namespace": "default", "id": "eb2b9e33-5d95-4666-9630-8fb8f8130dae", "state": "normal", "checkpoint_tso": 457114265808535554, "checkpoint_time": "2025-04-04 15:42:19.858", "error": null }' { "upstream_id": 7489359990447564738, "namespace": "default", "id": "eb2b9e33-5d95-4666-9630-8fb8f8130dae", "state": "normal", "checkpoint_tso": 457114265808535554, "checkpoint_time": "2025-04-04 15:42:19.858", "error": null } ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"eb2b9e33-5d95-4666-9630-8fb8f8130dae",' '"state":' '"normal",' '"checkpoint_tso":' 457114265808535554, '"checkpoint_time":' '"2025-04-04' '15:42:19.858",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359990447564738, '"namespace":' '"default",' '"id":' '"eb2b9e33-5d95-4666-9630-8fb8f8130dae",' '"state":' '"normal",' '"checkpoint_tso":' 457114265808535554, '"checkpoint_time":' '"2025-04-04' '15:42:19.858",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359990447564738 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5342.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114265867255809 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114265867255809 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:21 CST 2025] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.53905392.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 182 100 182 0 0 10705 0 --:--:-- --:--:-- --:--:-- 10705 { "version": "v9.0.0-beta.1-10-ge8bfbe4ba", "git_hash": "e8bfbe4ba98e05140ab733ee3d08dd3b99918721", "id": "64d73255-9689-4ebb-b91b-e9ea13db9287", "pid": 3461, "is_owner": true }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:23 CST 2025] <<<<<< run test case cli_tls_with_auth success! >>>>>> run task successfully Changefeed remove successfully. ID: f93678ed-53bf-4b12-85e1-675dd37d6f13 CheckpointTs: 457114264288362504 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-10992?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7489359990447564738 457114266660503554 457114265808535554 457114264288362504 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:24 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f {"id":"7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc00130f3 7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f /tidb/cdc/default/default/upstream/7489360089931700139 {"id":7489360089931700139,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f {"id":"7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc00130f3 7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f /tidb/cdc/default/default/upstream/7489360089931700139 {"id":7489360089931700139,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f {"id":"7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc00130f3 7a6ef74a-4b0d-4557-a5b0-b5f8ab9c675f /tidb/cdc/default/default/upstream/7489360089931700139 {"id":7489360089931700139,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5448.out cli changefeed create --start-ts=457114265867255809 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-31270?protocol=canal-json&enable-tidb-extension=true' run task successfully Changefeed remove successfully. ID: eb2b9e33-5d95-4666-9630-8fb8f8130dae CheckpointTs: 457114266922647555 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-10992?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7489359990447564738 run task successfully Create changefeed successfully! ID: c5366f64-1861-4eb8-8e9c-0aac1f0c0a15 Info: {"upstream_id":7489360089931700139,"namespace":"default","id":"c5366f64-1861-4eb8-8e9c-0aac1f0c0a15","sink_uri":"file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-31270?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:25.159748123+08:00","start_ts":457114265867255809,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114265867255809,"checkpoint_ts":457114265867255809,"checkpoint_time":"2025-04-04 15:42:20.082"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + set +x + workdir=/tmp/tidb_cdc_test/ddl_puller_lag + sink_uri='file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-31270?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_puller_lag ++ date + echo '[Fri Apr 4 15:42:26 CST 2025] <<<<<< START storage consumer in ddl_puller_lag case >>>>>>' [Fri Apr 4 15:42:26 CST 2025] <<<<<< START storage consumer in ddl_puller_lag case >>>>>> + cd /tmp/tidb_cdc_test/ddl_puller_lag + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_puller_lag + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-31270?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:26 CST 2025] <<<<<< run test case gc_safepoint success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.16 secs (493848158 bytes/sec) [Pipeline] { [Pipeline] cache =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl [2025/04/04 15:41:31.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252308906001,\"CommitTs\":457114252308906002,\"RowID\":1,\"PhysicalTableID\":114,\"TableInfo\":{\"id\":114,\"name\":{\"O\":\"example1\",\"L\":\"example1\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252295798832,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example1\",\"TableID\":114,\"IsPartition\":false},\"Version\":457114252308905985,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114253173981228,\"HandleKey\":1}"] [2025/04/04 15:41:33.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252335382575,\"CommitTs\":457114252335382579,\"RowID\":1,\"PhysicalTableID\":117,\"TableInfo\":{\"id\":117,\"name\":{\"O\":\"example2\",\"L\":\"example2\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252335382557,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example2\",\"TableID\":117,\"IsPartition\":false},\"Version\":457114252335382560,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114253698269189,\"HandleKey\":1}"] [2025/04/04 15:41:35.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252374441997,\"CommitTs\":457114252374442000,\"RowID\":1,\"PhysicalTableID\":120,\"TableInfo\":{\"id\":120,\"name\":{\"O\":\"example3\",\"L\":\"example3\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":7,\"Flag\":128,\"Flen\":19,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252361334831,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example3\",\"TableID\":120,\"IsPartition\":false},\"Version\":457114252361334835,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"2023-04-19 11:48:00\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":36,\"SplitTxn\":false,\"ReplicatingTs\":457114254222557188,\"HandleKey\":1}"] [2025/04/04 15:41:37.895 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252400656419,\"CommitTs\":457114252400656423,\"RowID\":1,\"PhysicalTableID\":123,\"TableInfo\":{\"id\":123,\"name\":{\"O\":\"example4\",\"L\":\"example4\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252400656400,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example4\",\"TableID\":123,\"IsPartition\":false},\"Version\":457114252400656404,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":457114254773059589,\"HandleKey\":1}"] [2025/04/04 15:41:40.129 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252439977985,\"CommitTs\":457114252439977988,\"RowID\":1,\"PhysicalTableID\":126,\"TableInfo\":{\"id\":126,\"name\":{\"O\":\"example5\",\"L\":\"example5\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252426870816,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example5\",\"TableID\":126,\"IsPartition\":false},\"Version\":457114252426870819,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114255362883587,\"HandleKey\":1}"] [2025/04/04 15:41:42.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252466192385,\"CommitTs\":457114252466192388,\"RowID\":1,\"PhysicalTableID\":129,\"TableInfo\":{\"id\":129,\"name\":{\"O\":\"example6\",\"L\":\"example6\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252453085215,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example6\",\"TableID\":129,\"IsPartition\":false},\"Version\":457114252453085219,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114256057827333,\"HandleKey\":1}"] [2025/04/04 15:41:44.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252479299650,\"CommitTs\":457114252479299653,\"RowID\":1,\"PhysicalTableID\":132,\"TableInfo\":{\"id\":132,\"name\":{\"O\":\"example7\",\"L\":\"example7\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8\",\"Collate\":\"utf8_general_ci\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252479299614,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example7\",\"TableID\":132,\"IsPartition\":false},\"Version\":457114252479299618,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":457114256581853189,\"HandleKey\":1}"] [2025/04/04 15:41:46.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252505513989,\"CommitTs\":457114252505513993,\"RowID\":-1,\"PhysicalTableID\":136,\"TableInfo\":{\"id\":135,\"name\":{\"O\":\"example8\",\"L\":\"example8\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252492406838,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":136,\"name\":{\"O\":\"b0\",\"L\":\"b0\"},\"less_than\":[\"0\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":137,\"name\":{\"O\":\"b1\",\"L\":\"b1\"},\"less_than\":[\"MAXVALUE\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":2,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example8\",\"TableID\":135,\"IsPartition\":true},\"Version\":457114252492406846,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":-1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114257106141189,\"HandleKey\":-1}"] [2025/04/04 15:41:46.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252505513993,\"CommitTs\":457114252505513994,\"RowID\":1,\"PhysicalTableID\":137,\"TableInfo\":{\"id\":135,\"name\":{\"O\":\"example8\",\"L\":\"example8\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252492406838,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":136,\"name\":{\"O\":\"b0\",\"L\":\"b0\"},\"less_than\":[\"0\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":137,\"name\":{\"O\":\"b1\",\"L\":\"b1\"},\"less_than\":[\"MAXVALUE\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":2,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example8\",\"TableID\":135,\"IsPartition\":true},\"Version\":457114252492406846,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114257106141190,\"HandleKey\":1}"] [2025/04/04 15:41:48.794 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252531728390,\"CommitTs\":457114252531728391,\"RowID\":1,\"PhysicalTableID\":140,\"TableInfo\":{\"id\":140,\"name\":{\"O\":\"example9\",\"L\":\"example9\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252518621243,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example9\",\"TableID\":140,\"IsPartition\":false},\"Version\":457114252518621248,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":457114257630691334,\"HandleKey\":1}"] [2025/04/04 15:41:50.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252544835593,\"CommitTs\":457114252544835594,\"RowID\":1,\"PhysicalTableID\":143,\"TableInfo\":{\"id\":143,\"name\":{\"O\":\"example10\",\"L\":\"example10\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":5,\"Flag\":0,\"Flen\":22,\"Decimal\":-1,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252531728431,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example10\",\"TableID\":143,\"IsPartition\":false},\"Version\":457114252531728434,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":36,\"SplitTxn\":false,\"ReplicatingTs\":457114258168086533,\"HandleKey\":1}"] [2025/04/04 15:41:52.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252571050009,\"CommitTs\":457114252571050012,\"RowID\":1,\"PhysicalTableID\":146,\"TableInfo\":{\"id\":146,\"name\":{\"O\":\"example11\",\"L\":\"example11\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":4099,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":0,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252571049989,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example11\",\"TableID\":146,\"IsPartition\":false},\"Version\":457114252571049993,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114258679267333,\"HandleKey\":1}"] [2025/04/04 15:41:54.794 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252597264438,\"CommitTs\":457114252597264441,\"RowID\":1,\"PhysicalTableID\":149,\"TableInfo\":{\"id\":149,\"name\":{\"O\":\"example12\",\"L\":\"example12\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":251,\"Flag\":0,\"Flen\":4294967295,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252597264419,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example12\",\"TableID\":149,\"IsPartition\":false},\"Version\":457114252597264422,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":457114259216400389,\"HandleKey\":1}"] [2025/04/04 15:41:56.794 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252636585989,\"CommitTs\":457114252636585992,\"RowID\":1,\"PhysicalTableID\":152,\"TableInfo\":{\"id\":152,\"name\":{\"O\":\"example13\",\"L\":\"example13\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":247,\"Flag\":0,\"Flen\":1,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":[\"a\",\"b\",\"c\"],\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252623740971,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example13\",\"TableID\":152,\"IsPartition\":false},\"Version\":457114252623740977,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114259740688389,\"HandleKey\":1}"] [2025/04/04 15:41:58.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252662800404,\"CommitTs\":457114252662800407,\"RowID\":1,\"PhysicalTableID\":155,\"TableInfo\":{\"id\":155,\"name\":{\"O\":\"example14\",\"L\":\"example14\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252662800385,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example14\",\"TableID\":155,\"IsPartition\":false},\"Version\":457114252662800388,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":457114260264976389,\"HandleKey\":1}"] [2025/04/04 15:42:00.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252689014821,\"CommitTs\":457114252689014824,\"RowID\":1,\"PhysicalTableID\":158,\"TableInfo\":{\"id\":158,\"name\":{\"O\":\"example15\",\"L\":\"example15\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252689014801,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example15\",\"TableID\":158,\"IsPartition\":false},\"Version\":457114252689014805,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":457114260789264390,\"HandleKey\":1}"] [2025/04/04 15:42:02.794 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252715229231,\"CommitTs\":457114252715229234,\"RowID\":1,\"PhysicalTableID\":162,\"TableInfo\":{\"id\":161,\"name\":{\"O\":\"example16\",\"L\":\"example16\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252715229212,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`a`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":162,\"name\":{\"O\":\"p0\",\"L\":\"p0\"},\"less_than\":[\"6\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":163,\"name\":{\"O\":\"p1\",\"L\":\"p1\"},\"less_than\":[\"11\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":164,\"name\":{\"O\":\"p2\",\"L\":\"p2\"},\"less_than\":[\"21\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":3,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example16\",\"TableID\":161,\"IsPartition\":true},\"Version\":457114252715229216,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114261313552389,\"HandleKey\":1}"] [2025/04/04 15:42:04.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252728336408,\"CommitTs\":457114252728336409,\"RowID\":18,\"PhysicalTableID\":166,\"TableInfo\":{\"id\":166,\"name\":{\"O\":\"example17\",\"L\":\"example17\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252728336388,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example17\",\"TableID\":166,\"IsPartition\":false},\"Version\":457114252728336393,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":18}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114261837840389,\"HandleKey\":18}"] [2025/04/04 15:42:06.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252741443650,\"CommitTs\":457114252754550786,\"RowID\":1,\"PhysicalTableID\":170,\"TableInfo\":{\"id\":169,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252741443620,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":170,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":171,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":172,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":173,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":174,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":175,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":6,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":169,\"IsPartition\":true},\"Version\":457114252741443625,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114262362390534,\"HandleKey\":1}"] [2025/04/04 15:42:06.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252741443650,\"CommitTs\":457114252754550786,\"RowID\":1977,\"PhysicalTableID\":173,\"TableInfo\":{\"id\":169,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252741443620,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":170,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":171,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":172,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":173,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":174,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":175,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":6,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":169,\"IsPartition\":true},\"Version\":457114252741443625,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1977}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114262362390537,\"HandleKey\":1977}"] [2025/04/04 15:42:06.796 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252741443650,\"CommitTs\":457114252754550786,\"RowID\":1999,\"PhysicalTableID\":175,\"TableInfo\":{\"id\":169,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252741443620,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":170,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":171,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":172,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":173,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":174,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":175,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"states\":null,\"num\":6,\"ddl_state\":0},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":169,\"IsPartition\":true},\"Version\":457114252741443625,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1999}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114262362390539,\"HandleKey\":1999}"] [2025/04/04 15:42:12.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114252820086832,\"CommitTs\":457114252820086834,\"RowID\":2003,\"PhysicalTableID\":182,\"TableInfo\":{\"id\":169,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":null,\"constraint_info\":null,\"fk_info\":null,\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114252820086812,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":null,\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":177,\"name\":{\"O\":\"pBefore1960\",\"L\":\"pbefore1960\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":172,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":173,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":180,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":182,\"name\":{\"O\":\"p1990to2010\",\"L\":\"p1990to2010\"},\"less_than\":[\"2010\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":null,\"dropping_definitions\":null,\"states\":null,\"num\":5,\"ddl_state\":0,\"ddl_type\":1,\"ddl_expr\":\"`id`\"},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":16,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":169,\"IsPartition\":true},\"Version\":457114252820086816,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":2003}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114263934992390,\"HandleKey\":2003}"] [2025/04/04 15:42:16.795 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":457114253173981222,\"CommitTs\":457114253173981223,\"RowID\":1,\"PhysicalTableID\":190,\"TableInfo\":{\"id\":190,\"name\":{\"O\":\"example19\",\"L\":\"example19\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":457114253173981203,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":112,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example19\",\"TableID\":190,\"IsPartition\":false},\"Version\":457114253173981207,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":457114264983568389,\"HandleKey\":1}"] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:42:29 CST 2025] <<<<<< run test case lossy_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/debezium_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:34 CST 2025] <<<<<< run test case debezium_basic success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.05 secs (500475106 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:37 CST 2025] <<<<<< run test case kafka_simple_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_file_exists false + local all_should_exist=false + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2025-04-03/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in ${EXIST_FILES[@]} + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2025-04-02/10.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/1.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/2.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/3.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/4.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/5.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/6.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/7.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/8.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/9.data ']' + for f in ${CLEANED_FILES[@]} + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2025-04-02/10.data ']' + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/schema.sql 127.0.0.1 4000 + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/data.sql 127.0.0.1 4000 + run_storage_consumer /tmp/tidb_cdc_test/storage_cleanup 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '' + workdir=/tmp/tidb_cdc_test/storage_cleanup + sink_uri='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 4 15:42:37 CST 2025] <<<<<< START storage consumer in storage_cleanup case >>>>>>' [Fri Apr 4 15:42:37 CST 2025] <<<<<< START storage consumer in storage_cleanup case >>>>>> + cd /tmp/tidb_cdc_test/storage_cleanup + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_cleanup + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + sleep 8 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "workload"="core" "readallfields"="true" "scanproportion"="0" "updateproportion"="0" "mysql.host"="127.0.0.1" "insertproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "mysql.db"="sink_retry" "mysql.port"="4000" "recordcount"="10" "operationcount"="0" "threadcount"="2" "dotransactions"="false" "readproportion"="0" ********************************************** Run finished, takes 4.034431ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3719.8, Avg(us): 722, Min(us): 469, Max(us): 1734, 95th(us): 2000, 99th(us): 2000 check diff failed 1-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360190305789573 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01e9e4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:4106, start at 2025-04-04 15:42:39.242750185 +0800 CST m=+2.549244273 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:39.252 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:39.225 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:39.225 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360190305789573 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01e9e4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:4106, start at 2025-04-04 15:42:39.242750185 +0800 CST m=+2.549244273 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:39.252 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:39.225 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:39.225 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360190787350249 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01eb780005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:4172, start at 2025-04-04 15:42:39.335678312 +0800 CST m=+2.549174041 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:39.346 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:39.326 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:39.326 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:41 CST 2025] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360196651936868 TiDB Cluster ID. tikv_gc_leader_uuid 657ff020b3c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:4148, start at 2025-04-04 15:42:41.374075561 +0800 CST m=+2.339595631 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.382 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360196651936868 TiDB Cluster ID. tikv_gc_leader_uuid 657ff020b3c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:4148, start at 2025-04-04 15:42:41.374075561 +0800 CST m=+2.339595631 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.382 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360195981628544 TiDB Cluster ID. tikv_gc_leader_uuid 657ff020bf0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:4222, start at 2025-04-04 15:42:41.434329198 +0800 CST m=+2.326070128 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.468 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.459 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.459 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5411.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360200917145733 TiDB Cluster ID. tikv_gc_leader_uuid 657ff021298000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:3884, start at 2025-04-04 15:42:41.847714411 +0800 CST m=+2.244770141 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.855 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.830 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.830 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360200917145733 TiDB Cluster ID. tikv_gc_leader_uuid 657ff021298000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:3884, start at 2025-04-04 15:42:41.847714411 +0800 CST m=+2.244770141 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.855 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.830 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.830 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360201509739625 TiDB Cluster ID. tikv_gc_leader_uuid 657ff021428000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:3934, start at 2025-04-04 15:42:41.968331284 +0800 CST m=+2.296213494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:41.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:41.968 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:41.968 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later + set +x + tso='457114271996968961 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114271996968961 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:44 CST 2025] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.54525454.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:44 CST 2025] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/storage_csv_update Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Fri Apr 4 15:42:45 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.55015503.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + check_sync_diff /tmp/tidb_cdc_test/storage_cleanup /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/diff_config.toml 100 check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Fri Apr 4 15:42:45 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.52415243.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 5-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 7.96 secs (506211499 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:48 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89e6f979-55f2-49a5-971d-6080bf0ae875 {"id":"89e6f979-55f2-49a5-971d-6080bf0ae875","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc05c88e2 89e6f979-55f2-49a5-971d-6080bf0ae875 /tidb/cdc/default/default/upstream/7489360190305789573 {"id":7489360190305789573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89e6f979-55f2-49a5-971d-6080bf0ae875 {"id":"89e6f979-55f2-49a5-971d-6080bf0ae875","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc05c88e2 89e6f979-55f2-49a5-971d-6080bf0ae875 /tidb/cdc/default/default/upstream/7489360190305789573 {"id":7489360190305789573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89e6f979-55f2-49a5-971d-6080bf0ae875 {"id":"89e6f979-55f2-49a5-971d-6080bf0ae875","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc05c88e2 89e6f979-55f2-49a5-971d-6080bf0ae875 /tidb/cdc/default/default/upstream/7489360190305789573 {"id":7489360190305789573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5511.out cli changefeed create --start-ts=457114271996968961 '--sink-uri=file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22606?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: e3b82297-cb5e-48ff-9eeb-92b5af9a67b3 check diff failed 6-th time, retry later Info: {"upstream_id":7489360190305789573,"namespace":"default","id":"e3b82297-cb5e-48ff-9eeb-92b5af9a67b3","sink_uri":"file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22606?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:48.466116785+08:00","start_ts":457114271996968961,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114271996968961,"checkpoint_ts":457114271996968961,"checkpoint_time":"2025-04-04 15:42:43.465"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:48 CST 2025] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:48 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e09f9bd6-c9da-4b20-9bdc-854556ff55f9 {"id":"e09f9bd6-c9da-4b20-9bdc-854556ff55f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065aed4 e09f9bd6-c9da-4b20-9bdc-854556ff55f9 /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e09f9bd6-c9da-4b20-9bdc-854556ff55f9 {"id":"e09f9bd6-c9da-4b20-9bdc-854556ff55f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065aed4 e09f9bd6-c9da-4b20-9bdc-854556ff55f9 /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e09f9bd6-c9da-4b20-9bdc-854556ff55f9 {"id":"e09f9bd6-c9da-4b20-9bdc-854556ff55f9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752565} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065aed4 e09f9bd6-c9da-4b20-9bdc-854556ff55f9 /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_reconstruct + sink_uri='file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_reconstruct ++ date + echo '[Fri Apr 4 15:42:48 CST 2025] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>>' [Fri Apr 4 15:42:48 CST 2025] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_reconstruct + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_reconstruct + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:49 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05a1c035-e7ce-4810-a579-197ecaf596f0 {"id":"05a1c035-e7ce-4810-a579-197ecaf596f0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752566} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067dfd7 05a1c035-e7ce-4810-a579-197ecaf596f0 /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05a1c035-e7ce-4810-a579-197ecaf596f0 {"id":"05a1c035-e7ce-4810-a579-197ecaf596f0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752566} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067dfd7 05a1c035-e7ce-4810-a579-197ecaf596f0 /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05a1c035-e7ce-4810-a579-197ecaf596f0 {"id":"05a1c035-e7ce-4810-a579-197ecaf596f0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752566} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067dfd7 05a1c035-e7ce-4810-a579-197ecaf596f0 /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5308.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7489360200917145733,"namespace":"default","id":"ddl-manager","sink_uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:49.517798302+08:00","start_ts":457114273557774340,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114273557774340,"checkpoint_ts":457114273557774340,"checkpoint_time":"2025-04-04 15:42:49.419"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 4 15:42:49 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 4 15:42:49 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster ***************** properties ***************** "readallfields"="true" "readproportion"="0" "recordcount"="50" "mysql.user"="root" "operationcount"="0" "workload"="core" "requestdistribution"="uniform" "mysql.port"="4000" "updateproportion"="0" "mysql.host"="127.0.0.1" "threadcount"="4" "dotransactions"="false" "insertproportion"="0" "scanproportion"="0" "mysql.db"="changefeed_reconstruct" ********************************************** Run finished, takes 22.63143ms INSERT - Takes(s): 0.0, Count: 48, OPS: 2782.5, Avg(us): 1787, Min(us): 1194, Max(us): 5674, 95th(us): 6000, 99th(us): 6000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/drop_many_tables + sink_uri='file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22606?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/drop_many_tables ++ date + echo '[Fri Apr 4 15:42:49 CST 2025] <<<<<< START storage consumer in drop_many_tables case >>>>>>' [Fri Apr 4 15:42:49 CST 2025] <<<<<< START storage consumer in drop_many_tables case >>>>>> + cd /tmp/tidb_cdc_test/drop_many_tables + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/drop_many_tables + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22606?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table drop_tables.c not exists for 1-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/ddl_manager + sink_uri='file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_manager ++ date + echo '[Fri Apr 4 15:42:50 CST 2025] <<<<<< START storage consumer in ddl_manager case >>>>>>' [Fri Apr 4 15:42:50 CST 2025] <<<<<< START storage consumer in ddl_manager case >>>>>> + cd /tmp/tidb_cdc_test/ddl_manager + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_manager + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_manager/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 7-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360237826554979 TiDB Cluster ID. tikv_gc_leader_uuid 657ff028d900007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:4459, start at 2025-04-04 15:42:49.709731467 +0800 CST m=+2.335251910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:49.716 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:49.700 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:49.700 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360237826554979 TiDB Cluster ID. tikv_gc_leader_uuid 657ff028d900007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:4459, start at 2025-04-04 15:42:49.709731467 +0800 CST m=+2.335251910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:49.716 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:49.700 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:49.700 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360227531060008 TiDB Cluster ID. tikv_gc_leader_uuid 657ff028fe4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:4540, start at 2025-04-04 15:42:49.889059532 +0800 CST m=+2.409973782 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:49.897 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:49.899 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:49.899 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:51 CST 2025] <<<<<< run test case kafka_simple_claim_check success! >>>>>> table changefeed_reconstruct.usertable not exists for 2-th check, retry later table drop_tables.c not exists for 2-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table changefeed_reconstruct.usertable not exists for 3-th check, retry later wait process cdc.test exit for 2-th time... [Fri Apr 4 15:42:53 CST 2025] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.59095911.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360247445609330 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02b85c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:5154, start at 2025-04-04 15:42:52.467857194 +0800 CST m=+2.629240754 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.477 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.490 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.490 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360247445609330 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02b85c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:5154, start at 2025-04-04 15:42:52.467857194 +0800 CST m=+2.629240754 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.477 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.490 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.490 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360243992211009 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02b9e00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:5229, start at 2025-04-04 15:42:52.540644998 +0800 CST m=+2.604516954 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.550 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.536 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.536 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:54 CST 2025] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> table drop_tables.c not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:55 CST 2025] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> table changefeed_reconstruct.usertable not exists for 4-th check, retry later [Fri Apr 4 15:42:56 CST 2025] <<<<<< START cdc server in storage_csv_update case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.65396541.out server --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_csv_update/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table drop_tables.c not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:56 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a69ed0b-38b8-44d8-af9a-dcbda475a15a {"id":"7a69ed0b-38b8-44d8-af9a-dcbda475a15a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752574} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08450d6 7a69ed0b-38b8-44d8-af9a-dcbda475a15a /tidb/cdc/default/default/upstream/7489360237826554979 {"id":7489360237826554979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a69ed0b-38b8-44d8-af9a-dcbda475a15a {"id":"7a69ed0b-38b8-44d8-af9a-dcbda475a15a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752574} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08450d6 7a69ed0b-38b8-44d8-af9a-dcbda475a15a /tidb/cdc/default/default/upstream/7489360237826554979 {"id":7489360237826554979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a69ed0b-38b8-44d8-af9a-dcbda475a15a {"id":"7a69ed0b-38b8-44d8-af9a-dcbda475a15a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752574} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc08450d6 7a69ed0b-38b8-44d8-af9a-dcbda475a15a /tidb/cdc/default/default/upstream/7489360237826554979 {"id":7489360237826554979,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_pause_resume + sink_uri='file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-21438?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_pause_resume ++ date + echo '[Fri Apr 4 15:42:57 CST 2025] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>>' [Fri Apr 4 15:42:57 CST 2025] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_pause_resume + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_pause_resume + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-21438?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 8.10 secs (497845358 bytes/sec) [Pipeline] { [Pipeline] cache table changefeed_pause_resume.t1 not exists for 1-th check, retry later table changefeed_reconstruct.usertable not exists for 5-th check, retry later table drop_tables.c not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:42:58 CST 2025] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:59 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3344a60e-77eb-4a13-a342-bef7cac56410 {"id":"3344a60e-77eb-4a13-a342-bef7cac56410","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752576} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc091c0d7 3344a60e-77eb-4a13-a342-bef7cac56410 /tidb/cdc/default/default/upstream/7489360247445609330 {"id":7489360247445609330,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3344a60e-77eb-4a13-a342-bef7cac56410 {"id":"3344a60e-77eb-4a13-a342-bef7cac56410","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752576} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc091c0d7 3344a60e-77eb-4a13-a342-bef7cac56410 /tidb/cdc/default/default/upstream/7489360247445609330 {"id":7489360247445609330,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3344a60e-77eb-4a13-a342-bef7cac56410 {"id":"3344a60e-77eb-4a13-a342-bef7cac56410","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752576} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc091c0d7 3344a60e-77eb-4a13-a342-bef7cac56410 /tidb/cdc/default/default/upstream/7489360247445609330 {"id":7489360247445609330,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.cli.6595.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml table changefeed_pause_resume.t1 not exists for 2-th check, retry later Create changefeed successfully! ID: 353b5da3-68b6-4bee-b80c-de131b568fb2 Info: {"upstream_id":7489360247445609330,"namespace":"default","id":"353b5da3-68b6-4bee-b80c-de131b568fb2","sink_uri":"file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:59.812433711+08:00","start_ts":457114276263100417,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114276263100417,"checkpoint_ts":457114276263100417,"checkpoint_time":"2025-04-04 15:42:59.739"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table changefeed_reconstruct.usertable exists check diff successfully table drop_tables.c not exists for 6-th check, retry later wait process 5506 exit for 1-th time... wait process 5506 exit for 2-th time... + set +x /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5506) - No such process wait process 5506 exit for 3-th time... process 5506 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:43:01 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.57795781.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL table changefeed_pause_resume.t1 not exists for 3-th check, retry later * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:01 CST 2025] <<<<<< run test case canal_json_basic success! >>>>>> table drop_tables.c not exists for 7-th check, retry later table changefeed_pause_resume.t1 not exists for 4-th check, retry later table drop_tables.c not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:04 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/421096f3-f03b-4e5d-8095-1e901414db50 {UpstreamID:7489360196651936868 Namespace:default ID:421096f3-f03b-4e5d-8095-1e901414db50 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:48.706777691 +0800 CST StartTs:457114273345437699 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00315ccf0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273358544903} {CheckpointTs:457114276098211842 MinTableBarrierTs:457114277408669699 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114276098211842, checkpointTs: 457114276098211842, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c4b5fd1-a702-449f-90f4-f7838f39a3cf {"id":"9c4b5fd1-a702-449f-90f4-f7838f39a3cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752582} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065afae 9c4b5fd1-a702-449f-90f4-f7838f39a3cf /tidb/cdc/default/default/changefeed/info/421096f3-f03b-4e5d-8095-1e901414db50 {"upstream-id":7489360196651936868,"namespace":"default","changefeed-id":"421096f3-f03b-4e5d-8095-1e901414db50","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:48.706777691+08:00","start-ts":457114273345437699,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273358544903} /tidb/cdc/default/default/changefeed/status/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":457114276098211842,"min-table-barrier-ts":457114277408669699,"admin-job-type":0} /tidb/cdc/default/default/task/position/9c4b5fd1-a702-449f-90f4-f7838f39a3cf/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/421096f3-f03b-4e5d-8095-1e901414db50 {UpstreamID:7489360196651936868 Namespace:default ID:421096f3-f03b-4e5d-8095-1e901414db50 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:48.706777691 +0800 CST StartTs:457114273345437699 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00315ccf0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273358544903} {CheckpointTs:457114276098211842 MinTableBarrierTs:457114277408669699 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114276098211842, checkpointTs: 457114276098211842, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c4b5fd1-a702-449f-90f4-f7838f39a3cf {"id":"9c4b5fd1-a702-449f-90f4-f7838f39a3cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752582} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065afae 9c4b5fd1-a702-449f-90f4-f7838f39a3cf /tidb/cdc/default/default/changefeed/info/421096f3-f03b-4e5d-8095-1e901414db50 {"upstream-id":7489360196651936868,"namespace":"default","changefeed-id":"421096f3-f03b-4e5d-8095-1e901414db50","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:48.706777691+08:00","start-ts":457114273345437699,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273358544903} /tidb/cdc/default/default/changefeed/status/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":457114276098211842,"min-table-barrier-ts":457114277408669699,"admin-job-type":0} /tidb/cdc/default/default/task/position/9c4b5fd1-a702-449f-90f4-f7838f39a3cf/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/421096f3-f03b-4e5d-8095-1e901414db50 {UpstreamID:7489360196651936868 Namespace:default ID:421096f3-f03b-4e5d-8095-1e901414db50 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:48.706777691 +0800 CST StartTs:457114273345437699 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00315ccf0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273358544903} {CheckpointTs:457114276098211842 MinTableBarrierTs:457114277408669699 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114276098211842, checkpointTs: 457114276098211842, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c4b5fd1-a702-449f-90f4-f7838f39a3cf {"id":"9c4b5fd1-a702-449f-90f4-f7838f39a3cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752582} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc065afae 9c4b5fd1-a702-449f-90f4-f7838f39a3cf /tidb/cdc/default/default/changefeed/info/421096f3-f03b-4e5d-8095-1e901414db50 {"upstream-id":7489360196651936868,"namespace":"default","changefeed-id":"421096f3-f03b-4e5d-8095-1e901414db50","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-10746?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:48.706777691+08:00","start-ts":457114273345437699,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273358544903} /tidb/cdc/default/default/changefeed/status/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":457114276098211842,"min-table-barrier-ts":457114277408669699,"admin-job-type":0} /tidb/cdc/default/default/task/position/9c4b5fd1-a702-449f-90f4-f7838f39a3cf/421096f3-f03b-4e5d-8095-1e901414db50 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360196651936868 {"id":7489360196651936868,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id table changefeed_pause_resume.t1 not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc) 4030545920 bytes in 6.84 secs (589174902 bytes/sec) [Pipeline] { [Pipeline] sh =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:05 CST 2025] <<<<<< run test case canal_json_content_compatible success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G16 [Pipeline] sh "id": "9c4b5fd1-a702-449f-90f4-f7838f39a3cf", "cluster-id": "default" run task successfully table drop_tables.c not exists for 9-th check, retry later Run cases: owner_resign processor_etcd_worker_delay sink_hang GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-5q5v3 HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=e1e8d38b-6bba-445e-b1c0-320e9bb457db GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G16 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz pingcap_tiflow_pull_cdc_integration_storage_test_3706-5q5v3 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G17 [Pipeline] sh =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:06 CST 2025] <<<<<< run test case cdc success! >>>>>> Run cases: clustered_index processor_resolved_ts_fallback GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-55kv6 HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=362c7dd6-7bde-48c2-855d-539bac971033 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G17 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-55kv6 pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:06 CST 2025] <<<<<< skip test case clustered_index for storage! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G14 [Pipeline] sh Run cases: changefeed_finish force_replicate_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-hfdj1 HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=26c0a004-2a82-43ff-b47b-413d2e9eea19 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G14 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp pingcap_tiflow_pull_cdc_integration_storage_test_3706-hfdj1 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G12 [Pipeline] sh table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 exists Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-tvh3r HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=1cd3d3ec-f08a-4648-a3c1-a7055db39eba GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G12 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-tvh3r pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:07 CST 2025] <<<<<< run test case many_pk_or_uk success! >>>>>> + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G15 [Pipeline] sh Run cases: new_ci_collation batch_add_table multi_rocks ci_collation_compatibility GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-lrnft HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=7b5582c3-02b6-4beb-a924-0160ee4febd2 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G15 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs pingcap_tiflow_pull_cdc_integration_storage_test_3706-lrnft PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:07 CST 2025] <<<<<< run test case new_ci_collation success! >>>>>> + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G13 capture_id: 9c4b5fd1-a702-449f-90f4-f7838f39a3cf check_processor_table_count http://127.0.0.1:2379 421096f3-f03b-4e5d-8095-1e901414db50 9c4b5fd1-a702-449f-90f4-f7838f39a3cf 1 run task successfully [Pipeline] sh Run cases: tiflash region_merge common_1 GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-mrmkt HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=2f53624e-b33f-4fbd-8092-4870121dcb65 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G13 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv pingcap_tiflow_pull_cdc_integration_storage_test_3706-mrmkt PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G11 check diff failed 1-th time, retry later Run cases: resolve_lock move_table autorandom generate_column GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-42jcc HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=6975387a-91f0-4545-be91-9baa3b929365 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G11 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-42jcc pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: storage... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G10 check_processor_table_count http://127.0.0.1:2379 421096f3-f03b-4e5d-8095-1e901414db50 9c4b5fd1-a702-449f-90f4-f7838f39a3cf 0 table drop_tables.c not exists for 10-th check, retry later Run cases: default_value simple cdc_server_tips event_filter sql_mode GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_storage_test","buildid":"1908057860684648448","prowjobid":"24af8db9-14d4-4a11-a066-a153aa2d09ac","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648448 POD_LABEL=pingcap_tiflow_pull_cdc_integration_storage_test_3706-5zfwb HOSTNAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=changes JENKINS_NODE_COOKIE=18b16422-69d8-4d31-8460-a141507a2627 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_storage_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3706 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_storage_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/display/redirect PROW_JOB_ID=24af8db9-14d4-4a11-a066-a153aa2d09ac TEST_GROUP=G10 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/3706/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_storage_test/ BUILD_DISPLAY_NAME=#3706 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_storage_test_3706-5zfwb pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-4a523405106a7d00374130edbe2665c6e1f4e192f859497a5647eb99843c7c0e _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: storage... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table count 1 does equal to expected count 0 run task failed 1-th time, retry later wait process 5246 exit for 1-th time... wait process 5246 exit for 2-th time... wait process 5246 exit for 3-th time... wait process 5246 exit for 4-th time... wait process 5246 exit for 5-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5246) - No such process wait process 5246 exit for 6-th time... process 5246 already exit [Fri Apr 4 15:43:08 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.54555457.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/csv_storage_basic Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:08 CST 2025] <<<<<< run test case multi_topics success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table drop_tables.c exists check diff successfully check diff failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check_processor_table_count http://127.0.0.1:2379 421096f3-f03b-4e5d-8095-1e901414db50 9c4b5fd1-a702-449f-90f4-f7838f39a3cf 0 run task successfully wait process cdc.test exit for 1-th time... table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:10 CST 2025] <<<<<< run test case sink_retry success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ***************** properties ***************** "mysql.port"="4000" "readproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "recordcount"="50" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_reconstruct" "updateproportion"="0" "scanproportion"="0" "workload"="core" "threadcount"="4" "insertproportion"="0" "readallfields"="true" "dotransactions"="false" "mysql.user"="root" ********************************************** Run finished, takes 20.092648ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3129.4, Avg(us): 1573, Min(us): 1034, Max(us): 5058, 95th(us): 5000, 99th(us): 6000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:12 CST 2025] <<<<<< run test case avro_basic success! >>>>>> The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... process cdc.test already exit [Fri Apr 4 15:43:12 CST 2025] <<<<<< run test case drop_many_tables success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... table changefeed_reconstruct.usertable not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:14 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3951 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274357313554,"min-table-barrier-ts":457114274357313554,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274357313554,"min-table-barrier-ts":457114274357313554,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274357313554,"min-table-barrier-ts":457114274357313554,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + break + set +x [Fri Apr 4 15:43:14 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.55175519.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:14 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3951 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274488385557,"min-table-barrier-ts":457114274488385557,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274488385557,"min-table-barrier-ts":457114274488385557,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489360200917145733 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:42:49.517798302 +0800 CST StartTs:457114273557774340 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004323050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114273570881544} {CheckpointTs:457114274357313554 MinTableBarrierTs:457114274357313554 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/263cce82-4ad2-4e4c-a72d-f04deff1d211 {"id":"263cce82-4ad2-4e4c-a72d-f04deff1d211","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752589} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc067e207 263cce82-4ad2-4e4c-a72d-f04deff1d211 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489360200917145733,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-25656?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:42:49.517798302+08:00","start-ts":457114273557774340,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114273570881544} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114274488385557,"min-table-barrier-ts":457114274488385557,"admin-job-type":0} /tidb/cdc/default/default/task/position/263cce82-4ad2-4e4c-a72d-f04deff1d211/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360200917145733 {"id":7489360200917145733,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 4-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:15 CST 2025] <<<<<< run test case canal_json_handle_key_only success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_reconstruct.usertable not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 2-th check, retry later check diff failed 5-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360349540003796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0425d0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:5141, start at 2025-04-04 15:43:15.873048155 +0800 CST m=+2.268908111 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:15.881 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:15.878 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:15.878 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360349540003796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0425d0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:5141, start at 2025-04-04 15:43:15.873048155 +0800 CST m=+2.268908111 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:15.881 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:15.878 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:15.878 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360345698111289 TiDB Cluster ID. tikv_gc_leader_uuid 657ff04268c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:5215, start at 2025-04-04 15:43:15.879966967 +0800 CST m=+2.204433371 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:15.924 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:15.924 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:15.924 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table changefeed_reconstruct.usertable not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 3-th check, retry later check diff failed 6-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:18 CST 2025] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360354056934566 TiDB Cluster ID. tikv_gc_leader_uuid 657ff043e94000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:1340, start at 2025-04-04 15:43:17.432174883 +0800 CST m=+2.427496905 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.440 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:17.413 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:17.413 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360354056934566 TiDB Cluster ID. tikv_gc_leader_uuid 657ff043e94000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:1340, start at 2025-04-04 15:43:17.432174883 +0800 CST m=+2.427496905 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.440 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:17.413 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:17.413 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360354315264920 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0441a8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:1406, start at 2025-04-04 15:43:17.646089353 +0800 CST m=+2.527020137 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.654 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:17.660 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:17.660 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360363620393784 TiDB Cluster ID. tikv_gc_leader_uuid 657ff045bc8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:1225, start at 2025-04-04 15:43:19.323036985 +0800 CST m=+2.008356328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.330 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360363620393784 TiDB Cluster ID. tikv_gc_leader_uuid 657ff045bc8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:1225, start at 2025-04-04 15:43:19.323036985 +0800 CST m=+2.008356328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.330 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:43:20 CST 2025] <<<<<< START cdc server in csv_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.65886590.out server --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_basic/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_reconstruct.usertable exists check diff successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360357851472828 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0456700006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz, pid:1238, start at 2025-04-04 15:43:18.946924368 +0800 CST m=+2.630222056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:18.955 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:18.940 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:18.940 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360357851472828 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0456700006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz, pid:1238, start at 2025-04-04 15:43:18.946924368 +0800 CST m=+2.630222056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:18.955 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:18.940 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:18.940 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360359391337583 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0458b80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-55k-0m5dz, pid:1300, start at 2025-04-04 15:43:19.089680472 +0800 CST m=+2.644021717 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.097 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.086 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.086 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:43:21 CST 2025] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.26332635.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360362934689504 TiDB Cluster ID. tikv_gc_leader_uuid 657ff045ac00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:1284, start at 2025-04-04 15:43:19.223604694 +0800 CST m=+2.228820633 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.231 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360362934689504 TiDB Cluster ID. tikv_gc_leader_uuid 657ff045ac00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:1284, start at 2025-04-04 15:43:19.223604694 +0800 CST m=+2.228820633 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.231 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360366858474825 TiDB Cluster ID. tikv_gc_leader_uuid 657ff045d140004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:1360, start at 2025-04-04 15:43:19.370080201 +0800 CST m=+2.298330629 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.377 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.365 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.365 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360366310171585 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0460a00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:1356, start at 2025-04-04 15:43:19.600903164 +0800 CST m=+2.242876033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.610 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.592 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.592 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360366310171585 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0460a00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:1356, start at 2025-04-04 15:43:19.600903164 +0800 CST m=+2.242876033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.610 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.592 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.592 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360357327176544 TiDB Cluster ID. tikv_gc_leader_uuid 657ff04611c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:1421, start at 2025-04-04 15:43:19.640627749 +0800 CST m=+2.209399303 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.649 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.692 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.692 +0800 All versions after safe point can be accessed. (DO NOT EDIT) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:21 CST 2025] <<<<<< run test case changefeed_reconstruct success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360361523375025 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0461280006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:1296, start at 2025-04-04 15:43:19.633707915 +0800 CST m=+2.222766128 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:19.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:19.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:19.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360369757898772 TiDB Cluster ID. tikv_gc_leader_uuid 657ff047cbc0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:1347, start at 2025-04-04 15:43:21.399345965 +0800 CST m=+2.333085056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.408 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.391 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.391 +0800 All versions after safe point can be accessed. (DO NOT EDIT) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:22 CST 2025] <<<<<< run test case canal_json_claim_check success! >>>>>> [Fri Apr 4 15:43:22 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.26652667.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:23 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 {"id":"063e97d3-dadc-4dfb-aad7-8ac30e4d17d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752600} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0eeabd6 063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 /tidb/cdc/default/default/upstream/7489360349540003796 {"id":7489360349540003796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 {"id":"063e97d3-dadc-4dfb-aad7-8ac30e4d17d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752600} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0eeabd6 063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 /tidb/cdc/default/default/upstream/7489360349540003796 {"id":7489360349540003796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 {"id":"063e97d3-dadc-4dfb-aad7-8ac30e4d17d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752600} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0eeabd6 063e97d3-dadc-4dfb-aad7-8ac30e4d17d0 /tidb/cdc/default/default/upstream/7489360349540003796 {"id":7489360349540003796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.cli.6640.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360369757898772 TiDB Cluster ID. tikv_gc_leader_uuid 657ff047cbc0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:1347, start at 2025-04-04 15:43:21.399345965 +0800 CST m=+2.333085056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.408 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.391 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.391 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360370043889204 TiDB Cluster ID. tikv_gc_leader_uuid 657ff047e4c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:1414, start at 2025-04-04 15:43:21.512934886 +0800 CST m=+2.348910984 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.520 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.491 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.491 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 8-th time, retry later Create changefeed successfully! ID: 76f9cd30-aafa-478d-aaab-d9a47b173549 Info: {"upstream_id":7489360349540003796,"namespace":"default","id":"76f9cd30-aafa-478d-aaab-d9a47b173549","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s","create_time":"2025-04-04T15:43:23.534215262+08:00","start_ts":457114282473029637,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114282473029637,"checkpoint_ts":457114282473029637,"checkpoint_time":"2025-04-04 15:43:23.428"} PASS =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360374880194642 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0481440006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:4370, start at 2025-04-04 15:43:21.689149591 +0800 CST m=+2.418703071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.698 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.681 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.681 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360374880194642 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0481440006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:4370, start at 2025-04-04 15:43:21.689149591 +0800 CST m=+2.418703071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.698 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.681 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.681 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360372922866710 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0484540007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-z2l-q6d4s, pid:4437, start at 2025-04-04 15:43:21.88956219 +0800 CST m=+2.522772908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:21.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:21.877 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:21.877 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:43:23 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.26622664.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2677.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2668.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:24 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 {"id":"0c892ec5-57fd-4a0b-8f41-0fcdd27229a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f609d3 0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 /tidb/cdc/default/default/upstream/7489360354056934566 {"id":7489360354056934566,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 {"id":"0c892ec5-57fd-4a0b-8f41-0fcdd27229a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f609d3 0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 /tidb/cdc/default/default/upstream/7489360354056934566 {"id":7489360354056934566,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 {"id":"0c892ec5-57fd-4a0b-8f41-0fcdd27229a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752601} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f609d3 0c892ec5-57fd-4a0b-8f41-0fcdd27229a4 /tidb/cdc/default/default/upstream/7489360354056934566 {"id":7489360354056934566,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 6-th check, retry later + workdir=/tmp/tidb_cdc_test/changefeed_finish + sink_uri='file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-5948?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_finish ++ date + echo '[Fri Apr 4 15:43:24 CST 2025] <<<<<< START storage consumer in changefeed_finish case >>>>>>' [Fri Apr 4 15:43:24 CST 2025] <<<<<< START storage consumer in changefeed_finish case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_finish + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_finish + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-5948?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360379807276299 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0490100001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:1295, start at 2025-04-04 15:43:22.629306305 +0800 CST m=+2.191491359 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:22.636 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:22.628 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:22.628 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360379807276299 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0490100001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:1295, start at 2025-04-04 15:43:22.629306305 +0800 CST m=+2.191491359 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:22.636 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:22.628 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:22.628 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360376731773768 TiDB Cluster ID. tikv_gc_leader_uuid 657ff049144000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:1384, start at 2025-04-04 15:43:22.742564693 +0800 CST m=+2.194344358 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:22.749 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:22.755 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:22.755 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_basic ++ date + echo '[Fri Apr 4 15:43:25 CST 2025] <<<<<< START storage consumer in csv_storage_basic case >>>>>>' [Fri Apr 4 15:43:25 CST 2025] <<<<<< START storage consumer in csv_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360374070982322 TiDB Cluster ID. tikv_gc_leader_uuid 657ff049a70000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:1293, start at 2025-04-04 15:43:23.308673955 +0800 CST m=+2.170173368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:23.316 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:23.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:23.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360374070982322 TiDB Cluster ID. tikv_gc_leader_uuid 657ff049a70000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:1293, start at 2025-04-04 15:43:23.308673955 +0800 CST m=+2.170173368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:23.316 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:23.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:23.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360381858083152 TiDB Cluster ID. tikv_gc_leader_uuid 657ff049dd8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:1363, start at 2025-04-04 15:43:23.530267321 +0800 CST m=+2.310745608 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:23.538 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:23.510 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:23.510 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 9-th time, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114282642636801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114282642636801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:25 CST 2025] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.27112713.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5679.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114282642112513 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114282642112513 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.2732.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:43:25 CST 2025] <<<<<< run test case open_protocol_claim_check success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:25 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.2716.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: b958501b-a17d-4c92-959e-c22f7ed2e90c Info: {"upstream_id":7489360357851472828,"namespace":"default","id":"b958501b-a17d-4c92-959e-c22f7ed2e90c","sink_uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:26.263221935+08:00","start_ts":457114283197071364,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114283197071364,"checkpoint_ts":457114283197071364,"checkpoint_time":"2025-04-04 15:43:26.190"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:43:26 CST 2025] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.27162718.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:26 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2750.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 [Fri Apr 4 15:43:26 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.27652767.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 7-th check, retry later Create changefeed successfully! ID: ec138267-86e7-4420-a672-a9c58bddc753 Info: {"upstream_id":7489360362934689504,"namespace":"default","id":"ec138267-86e7-4420-a672-a9c58bddc753","sink_uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:27.04954115+08:00","start_ts":457114283400757253,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114283400757253,"checkpoint_ts":457114283400757253,"checkpoint_time":"2025-04-04 15:43:26.967"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114283155390465 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114283155390465 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 2-th time, retry later + set +x + tso='457114283144904705 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114283144904705 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:27 CST 2025] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.27662768.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/processor_resolved_ts_fallback + sink_uri='file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_resolved_ts_fallback ++ date + echo '[Fri Apr 4 15:43:27 CST 2025] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>>' [Fri Apr 4 15:43:27 CST 2025] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>> + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 10-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2756.out cli tso query --pd=http://127.0.0.1:2379 + set +x + workdir=/tmp/tidb_cdc_test/owner_resign + sink_uri='file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/owner_resign ++ date + echo '[Fri Apr 4 15:43:28 CST 2025] <<<<<< START storage consumer in owner_resign case >>>>>>' [Fri Apr 4 15:43:28 CST 2025] <<<<<< START storage consumer in owner_resign case >>>>>> + cd /tmp/tidb_cdc_test/owner_resign + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/owner_resign + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/owner_resign/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:28 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce78bbd9-8d83-4eec-bc36-2a8786796363 {"id":"ce78bbd9-8d83-4eec-bc36-2a8786796363","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752605} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f936e3 ce78bbd9-8d83-4eec-bc36-2a8786796363 /tidb/cdc/default/default/upstream/7489360363620393784 {"id":7489360363620393784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce78bbd9-8d83-4eec-bc36-2a8786796363 {"id":"ce78bbd9-8d83-4eec-bc36-2a8786796363","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752605} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f936e3 ce78bbd9-8d83-4eec-bc36-2a8786796363 /tidb/cdc/default/default/upstream/7489360363620393784 {"id":7489360363620393784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce78bbd9-8d83-4eec-bc36-2a8786796363 {"id":"ce78bbd9-8d83-4eec-bc36-2a8786796363","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752605} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f936e3 ce78bbd9-8d83-4eec-bc36-2a8786796363 /tidb/cdc/default/default/upstream/7489360363620393784 {"id":7489360363620393784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: c31546ed-12b1-46c8-a4eb-ff7fd871efa1 Info: {"upstream_id":7489360363620393784,"namespace":"default","id":"c31546ed-12b1-46c8-a4eb-ff7fd871efa1","sink_uri":"file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-17154?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:28.74676415+08:00","start_ts":457114282642636801,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114282642636801,"checkpoint_ts":457114282642636801,"checkpoint_time":"2025-04-04 15:43:24.075"} + workdir=/tmp/tidb_cdc_test/tiflash + sink_uri='file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-17154?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/tiflash ++ date + echo '[Fri Apr 4 15:43:28 CST 2025] <<<<<< START storage consumer in tiflash case >>>>>>' [Fri Apr 4 15:43:28 CST 2025] <<<<<< START storage consumer in tiflash case >>>>>> + cd /tmp/tidb_cdc_test/tiflash + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/tiflash + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/tiflash/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-17154?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later run task failed 1-th time, retry later 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 8-th check, retry later + set +x + tso='457114283635638273 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114283635638273 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:29 CST 2025] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.27912793.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:43:29 CST 2025] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.57225724.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:29 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bd25945-a017-4deb-96de-aaaf65c5fb81 {"id":"1bd25945-a017-4deb-96de-aaaf65c5fb81","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752606} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fb5aff 1bd25945-a017-4deb-96de-aaaf65c5fb81 /tidb/cdc/default/default/upstream/7489360366310171585 {"id":7489360366310171585,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bd25945-a017-4deb-96de-aaaf65c5fb81 {"id":"1bd25945-a017-4deb-96de-aaaf65c5fb81","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752606} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fb5aff 1bd25945-a017-4deb-96de-aaaf65c5fb81 /tidb/cdc/default/default/upstream/7489360366310171585 {"id":7489360366310171585,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bd25945-a017-4deb-96de-aaaf65c5fb81 {"id":"1bd25945-a017-4deb-96de-aaaf65c5fb81","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752606} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fb5aff 1bd25945-a017-4deb-96de-aaaf65c5fb81 /tidb/cdc/default/default/upstream/7489360366310171585 {"id":7489360366310171585,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2773.out cli changefeed create --start-ts=457114282642112513 '--sink-uri=file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-29909?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 90bf50c5-f917-4b6f-9d26-aa8b730ed26c Info: {"upstream_id":7489360366310171585,"namespace":"default","id":"90bf50c5-f917-4b6f-9d26-aa8b730ed26c","sink_uri":"file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-29909?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:30.057507853+08:00","start_ts":457114282642112513,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114282642112513,"checkpoint_ts":457114282642112513,"checkpoint_time":"2025-04-04 15:43:24.073"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:30 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3645d222-6351-4c0b-9afe-00bd0b91cd00 {"id":"3645d222-6351-4c0b-9afe-00bd0b91cd00","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1056dd9 3645d222-6351-4c0b-9afe-00bd0b91cd00 /tidb/cdc/default/default/upstream/7489360379807276299 {"id":7489360379807276299,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3645d222-6351-4c0b-9afe-00bd0b91cd00 {"id":"3645d222-6351-4c0b-9afe-00bd0b91cd00","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1056dd9 3645d222-6351-4c0b-9afe-00bd0b91cd00 /tidb/cdc/default/default/upstream/7489360379807276299 {"id":7489360379807276299,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3645d222-6351-4c0b-9afe-00bd0b91cd00 {"id":"3645d222-6351-4c0b-9afe-00bd0b91cd00","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1056dd9 3645d222-6351-4c0b-9afe-00bd0b91cd00 /tidb/cdc/default/default/upstream/7489360379807276299 {"id":7489360379807276299,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.2821.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-31667?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 9ed31710-cff0-4a88-987e-876713d9e19f Info: {"upstream_id":7489360379807276299,"namespace":"default","id":"9ed31710-cff0-4a88-987e-876713d9e19f","sink_uri":"file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-31667?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:30.386043519+08:00","start_ts":457114284275269637,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114284275269637,"checkpoint_ts":457114284275269637,"checkpoint_time":"2025-04-04 15:43:30.303"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' table processor_resolved_ts_fallback.t1 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 9-th check, retry later 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:30 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a3017de-2434-4320-9065-53defc584e0f {"id":"3a3017de-2434-4320-9065-53defc584e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc10404f6 3a3017de-2434-4320-9065-53defc584e0f /tidb/cdc/default/default/upstream/7489360369757898772 {"id":7489360369757898772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a3017de-2434-4320-9065-53defc584e0f {"id":"3a3017de-2434-4320-9065-53defc584e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc10404f6 3a3017de-2434-4320-9065-53defc584e0f /tidb/cdc/default/default/upstream/7489360369757898772 {"id":7489360369757898772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a3017de-2434-4320-9065-53defc584e0f {"id":"3a3017de-2434-4320-9065-53defc584e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752607} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc10404f6 3a3017de-2434-4320-9065-53defc584e0f /tidb/cdc/default/default/upstream/7489360369757898772 {"id":7489360369757898772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table cdc_tiflash_test.multi_data_type not exists for 2-th check, retry later check diff failed 4-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/simple + sink_uri='file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-29909?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/simple ++ date + echo '[Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in simple case >>>>>>' [Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in simple case >>>>>> + cd /tmp/tidb_cdc_test/simple + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/simple + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/simple/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-29909?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' succeed to verify meta placement rules + workdir=/tmp/tidb_cdc_test/capture_session_done_during_task + sink_uri='file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-26815?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/capture_session_done_during_task ++ date + echo '[Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>>' [Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>> + cd /tmp/tidb_cdc_test/capture_session_done_during_task + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/capture_session_done_during_task + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-26815?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/batch_add_table + sink_uri='file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-31667?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/batch_add_table ++ date + echo '[Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in batch_add_table case >>>>>>' [Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in batch_add_table case >>>>>> + cd /tmp/tidb_cdc_test/batch_add_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/batch_add_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/batch_add_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-31667?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' lease 223195ffc10404f6 revoked Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:32 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7c69dcb-c70f-453f-ac3f-33e00a229004 {"id":"d7c69dcb-c70f-453f-ac3f-33e00a229004","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc107ffe9 d7c69dcb-c70f-453f-ac3f-33e00a229004 /tidb/cdc/default/default/upstream/7489360374070982322 {"id":7489360374070982322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7c69dcb-c70f-453f-ac3f-33e00a229004 {"id":"d7c69dcb-c70f-453f-ac3f-33e00a229004","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc107ffe9 d7c69dcb-c70f-453f-ac3f-33e00a229004 /tidb/cdc/default/default/upstream/7489360374070982322 {"id":7489360374070982322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7c69dcb-c70f-453f-ac3f-33e00a229004 {"id":"d7c69dcb-c70f-453f-ac3f-33e00a229004","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc107ffe9 d7c69dcb-c70f-453f-ac3f-33e00a229004 /tidb/cdc/default/default/upstream/7489360374070982322 {"id":7489360374070982322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2847.out cli changefeed create --start-ts=457114283635638273 '--sink-uri=file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5680?protocol=canal-json&enable-tidb-extension=true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:32 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 {"id":"8d0e68e8-7687-4c8e-8ce3-75d990f8cc30","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fee609 8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 /tidb/cdc/default/default/upstream/7489360374880194642 {"id":7489360374880194642,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 {"id":"8d0e68e8-7687-4c8e-8ce3-75d990f8cc30","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fee609 8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 /tidb/cdc/default/default/upstream/7489360374880194642 {"id":7489360374880194642,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 {"id":"8d0e68e8-7687-4c8e-8ce3-75d990f8cc30","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752609} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0fee609 8d0e68e8-7687-4c8e-8ce3-75d990f8cc30 /tidb/cdc/default/default/upstream/7489360374880194642 {"id":7489360374880194642,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5778.out cli changefeed create --start-ts=457114283155390465 '--sink-uri=file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-15144?protocol=canal-json&enable-tidb-extension=true' --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml Create changefeed successfully! ID: f9275a7d-6802-494f-813e-319eeda7b8a9 Info: {"upstream_id":7489360374880194642,"namespace":"default","id":"f9275a7d-6802-494f-813e-319eeda7b8a9","sink_uri":"file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-15144?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:32.592821204+08:00","start_ts":457114283155390465,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114283155390465,"checkpoint_ts":457114283155390465,"checkpoint_time":"2025-04-04 15:43:26.031"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 10-th check, retry later table capture_session_done_during_task.t exists check diff failed 1-th time, retry later table owner_resign.t1 not exists for 2-th check, retry later Create changefeed successfully! ID: 4c9e441e-eb83-4696-b024-113ef5c65770 Info: {"upstream_id":7489360374070982322,"namespace":"default","id":"4c9e441e-eb83-4696-b024-113ef5c65770","sink_uri":"file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5680?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:32.764376276+08:00","start_ts":457114283635638273,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114283635638273,"checkpoint_ts":457114283635638273,"checkpoint_time":"2025-04-04 15:43:27.863"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_basic Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 1-th time, retry later check diff failed 2-th time, retry later ls /tmp/tidb_cdc_test/storage_csv_update/storage_test/test/test_update 457114276669685765 run task successfully start check update + workdir=/tmp/tidb_cdc_test/storage_csv_update + sink_uri='file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_csv_update ++ date + echo '[Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in storage_csv_update case >>>>>>' [Fri Apr 4 15:43:31 CST 2025] <<<<<< START storage consumer in storage_csv_update case >>>>>> + cd /tmp/tidb_cdc_test/storage_csv_update + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_csv_update + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml table cdc_tiflash_test.multi_data_type not exists for 3-th check, retry later table processor_resolved_ts_fallback.t1 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/split_region + sink_uri='file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-15144?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/split_region ++ date + echo '[Fri Apr 4 15:43:34 CST 2025] <<<<<< START storage consumer in split_region case >>>>>>' [Fri Apr 4 15:43:34 CST 2025] <<<<<< START storage consumer in split_region case >>>>>> + cd /tmp/tidb_cdc_test/split_region + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/split_region + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/split_region/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-15144?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table split_region.test1 not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 2-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 2-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/resolve_lock + sink_uri='file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5680?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resolve_lock ++ date + echo '[Fri Apr 4 15:43:34 CST 2025] <<<<<< START storage consumer in resolve_lock case >>>>>>' [Fri Apr 4 15:43:34 CST 2025] <<<<<< START storage consumer in resolve_lock case >>>>>> + cd /tmp/tidb_cdc_test/resolve_lock + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resolve_lock + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resolve_lock/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5680?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' go: downloading go1.23.4 (linux/amd64) table ddl_manager.finish_mark not exists for 11-th check, retry later check diff failed 3-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 4-th check, retry later table processor_resolved_ts_fallback.t1 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table owner_resign.t1 not exists for 3-th check, retry later check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360427031806945 TiDB Cluster ID. tikv_gc_leader_uuid 657ff053f3c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:3947, start at 2025-04-04 15:43:33.888032447 +0800 CST m=+2.223572291 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:33.934 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:33.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:33.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360427031806945 TiDB Cluster ID. tikv_gc_leader_uuid 657ff053f3c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:3947, start at 2025-04-04 15:43:33.888032447 +0800 CST m=+2.223572291 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:33.934 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:33.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:33.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360425221606559 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0542600004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:4022, start at 2025-04-04 15:43:34.046848177 +0800 CST m=+2.295127444 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:34.054 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:34.040 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:34.040 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff successfully Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table split_region.test1 not exists for 2-th check, retry later table batch_add_table.finish_mark not exists for 3-th check, retry later check diff failed 4-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_resolved_ts_fallback.t1 not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later check diff failed 3-th time, retry later table owner_resign.t1 not exists for 4-th check, retry later check diff successfully [Fri Apr 4 15:43:37 CST 2025] <<<<<< run test case csv_storage_basic success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 4-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table split_region.test1 not exists for 3-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5405.out cli tso query --pd=http://127.0.0.1:2379 table batch_add_table.finish_mark not exists for 4-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table cdc_tiflash_test.multi_data_type exists Verifying downstream PD is started... table processor_resolved_ts_fallback.t1 exists [Fri Apr 4 15:43:38 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.28632865.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 13-th check, retry later check diff failed 4-th time, retry later table owner_resign.t1 exists check diff failed 1-th time, retry later check diff successfully go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/pkg/errors v0.9.1 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/docker/go-units v0.5.0 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/coreos/go-semver v0.3.1 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 check diff failed 5-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360440687679498 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058bd4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:6655, start at 2025-04-04 15:43:38.758722011 +0800 CST m=+3.609460212 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.741 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.741 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360440687679498 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058bd4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:6655, start at 2025-04-04 15:43:38.758722011 +0800 CST m=+3.609460212 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.741 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.741 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360437309379030 TiDB Cluster ID. tikv_gc_leader_uuid 657ff058efc0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:6716, start at 2025-04-04 15:43:38.955873757 +0800 CST m=+3.703084398 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:38.965 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:38.943 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:38.943 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 check diff failed 1-th time, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 5-th time, retry later + set +x + tso='457114286408073217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114286408073217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading golang.org/x/net v0.33.0 ***************** properties ***************** "mysql.port"="4000" "operationcount"="0" "readallfields"="true" "recordcount"="20" "mysql.db"="changefeed_error" "updateproportion"="0" "mysql.host"="127.0.0.1" "scanproportion"="0" "dotransactions"="false" "mysql.user"="root" "threadcount"="4" "requestdistribution"="uniform" "insertproportion"="0" "readproportion"="0" "workload"="core" ********************************************** Run finished, takes 10.685048ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3438.0, Avg(us): 1995, Min(us): 1036, Max(us): 4853, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:43:39 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.54665468.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading golang.org/x/text v0.21.0 wait process cdc.test exit for 2-th time... table split_region.test1 not exists for 4-th check, retry later table batch_add_table.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:40 CST 2025] <<<<<< run test case tiflash success! >>>>>> table ddl_manager.finish_mark not exists for 14-th check, retry later check diff failed 6-th time, retry later check diff failed 2-th time, retry later check diff failed 5-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/cockroachdb/pebble v1.1.0 [Fri Apr 4 15:43:41 CST 2025] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.80068008.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table split_region.test1 not exists for 5-th check, retry later check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b958501b-a17d-4c92-959e-c22f7ed2e90c {UpstreamID:7489360357851472828 Namespace:default ID:b958501b-a17d-4c92-959e-c22f7ed2e90c SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:26.263221935 +0800 CST StartTs:457114283197071364 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e92a20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283209916422} {CheckpointTs:457114287181398019 MinTableBarrierTs:457114287181398020 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/4cc15f4a-2b11-4e37-8d49-5153372fcec8 {"id":"4cc15f4a-2b11-4e37-8d49-5153372fcec8","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f529af 4cc15f4a-2b11-4e37-8d49-5153372fcec8 /tidb/cdc/default/default/changefeed/info/b958501b-a17d-4c92-959e-c22f7ed2e90c {"upstream-id":7489360357851472828,"namespace":"default","changefeed-id":"b958501b-a17d-4c92-959e-c22f7ed2e90c","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:26.263221935+08:00","start-ts":457114283197071364,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283209916422} /tidb/cdc/default/default/changefeed/status/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":457114287181398019,"min-table-barrier-ts":457114287181398020,"admin-job-type":0} /tidb/cdc/default/default/task/position/178f1536-287f-4a3d-89e2-278eab2ffcc1/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4cc15f4a-2b11-4e37-8d49-5153372fcec8/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b958501b-a17d-4c92-959e-c22f7ed2e90c {UpstreamID:7489360357851472828 Namespace:default ID:b958501b-a17d-4c92-959e-c22f7ed2e90c SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:26.263221935 +0800 CST StartTs:457114283197071364 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e92a20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283209916422} {CheckpointTs:457114287181398019 MinTableBarrierTs:457114287181398020 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/4cc15f4a-2b11-4e37-8d49-5153372fcec8 {"id":"4cc15f4a-2b11-4e37-8d49-5153372fcec8","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f529af 4cc15f4a-2b11-4e37-8d49-5153372fcec8 /tidb/cdc/default/default/changefeed/info/b958501b-a17d-4c92-959e-c22f7ed2e90c {"upstream-id":7489360357851472828,"namespace":"default","changefeed-id":"b958501b-a17d-4c92-959e-c22f7ed2e90c","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:26.263221935+08:00","start-ts":457114283197071364,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283209916422} /tidb/cdc/default/default/changefeed/status/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":457114287181398019,"min-table-barrier-ts":457114287181398020,"admin-job-type":0} /tidb/cdc/default/default/task/position/178f1536-287f-4a3d-89e2-278eab2ffcc1/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4cc15f4a-2b11-4e37-8d49-5153372fcec8/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/b958501b-a17d-4c92-959e-c22f7ed2e90c {UpstreamID:7489360357851472828 Namespace:default ID:b958501b-a17d-4c92-959e-c22f7ed2e90c SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:26.263221935 +0800 CST StartTs:457114283197071364 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e92a20 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283209916422} {CheckpointTs:457114287181398019 MinTableBarrierTs:457114287181398020 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/178f1536-287f-4a3d-89e2-278eab2ffcc1 {"id":"178f1536-287f-4a3d-89e2-278eab2ffcc1","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/4cc15f4a-2b11-4e37-8d49-5153372fcec8 {"id":"4cc15f4a-2b11-4e37-8d49-5153372fcec8","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f528da 178f1536-287f-4a3d-89e2-278eab2ffcc1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f529af 4cc15f4a-2b11-4e37-8d49-5153372fcec8 /tidb/cdc/default/default/changefeed/info/b958501b-a17d-4c92-959e-c22f7ed2e90c {"upstream-id":7489360357851472828,"namespace":"default","changefeed-id":"b958501b-a17d-4c92-959e-c22f7ed2e90c","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-5920?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:26.263221935+08:00","start-ts":457114283197071364,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283209916422} /tidb/cdc/default/default/changefeed/status/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":457114287181398019,"min-table-barrier-ts":457114287181398020,"admin-job-type":0} /tidb/cdc/default/default/task/position/178f1536-287f-4a3d-89e2-278eab2ffcc1/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4cc15f4a-2b11-4e37-8d49-5153372fcec8/b958501b-a17d-4c92-959e-c22f7ed2e90c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357851472828 {"id":7489360357851472828,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table batch_add_table.finish_mark not exists for 6-th check, retry later check data successfully go: downloading golang.org/x/time v0.7.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 wait process cdc.test exit for 1-th time... go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.29.11 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 table ddl_manager.finish_mark not exists for 15-th check, retry later go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360454911571584 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05aba00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:6627, start at 2025-04-04 15:43:40.781788408 +0800 CST m=+2.340639685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.790 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.776 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.776 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360454911571584 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05aba00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:6627, start at 2025-04-04 15:43:40.781788408 +0800 CST m=+2.340639685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.790 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.776 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.776 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360445435255640 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05ade00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:6700, start at 2025-04-04 15:43:40.968700266 +0800 CST m=+2.460201003 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:40.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:40.970 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:40.970 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 wait process cdc.test exit for 2-th time... table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later check diff failed 7-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:43 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 {"id":"3cd2a78f-f33f-4cf6-b3cb-a51082e478b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752620} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1372ef3 3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 {"id":"3cd2a78f-f33f-4cf6-b3cb-a51082e478b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752620} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1372ef3 3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 {"id":"3cd2a78f-f33f-4cf6-b3cb-a51082e478b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752620} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1372ef3 3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5530.out cli changefeed create --start-ts=457114286408073217 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error check diff failed 3-th time, retry later check diff successfully check diff failed 1-th time, retry later Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7489360427031806945,"namespace":"default","id":"changefeed-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:43.480953984+08:00","start_ts":457114286408073217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217,"checkpoint_time":"2025-04-04 15:43:38.439"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:43 CST 2025] <<<<<< run test case simple success! >>>>>> go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/andybalholm/brotli v1.0.5 check diff successfully [Fri Apr 4 15:43:43 CST 2025] <<<<<< run test case storage_csv_update success! >>>>>> table split_region.test1 not exists for 6-th check, retry later go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading k8s.io/apimachinery v0.29.11 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b table ddl_manager.finish_mark not exists for 16-th check, retry later table batch_add_table.finish_mark not exists for 7-th check, retry later [Fri Apr 4 15:43:44 CST 2025] <<<<<< START cdc server in canal_json_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.79537955.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/changefeed_error + sink_uri='file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_error ++ date + echo '[Fri Apr 4 15:43:44 CST 2025] <<<<<< START storage consumer in changefeed_error case >>>>>>' [Fri Apr 4 15:43:44 CST 2025] <<<<<< START storage consumer in changefeed_error case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_error + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_error + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_error/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:43:43.622139633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at 457114286408073217" } }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:43:43.622139633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at 457114286408073217" } }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:43:43.622139633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at 457114286408073217" } } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114286408073217, '"checkpoint_time":' '"2025-04-04' '15:43:38.439",' '"error":' '{' '"time":' '"2025-04-04T15:43:43.622139633+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at '457114286408073217"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114286408073217, '"checkpoint_time":' '"2025-04-04' '15:43:38.439",' '"error":' '{' '"time":' '"2025-04-04T15:43:43.622139633+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at '457114286408073217"' '}' '}' ++ jq -r .error.message table processor_resolved_ts_fallback.t2 not exists for 2-th check, retry later + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at 457114286408073217' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114286408072917 is earlier than or equal to GC safepoint at 457114286408073217 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5643.out cli changefeed resume -c changefeed-error go: downloading github.com/pierrec/lz4 v2.6.1+incompatible check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:44 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9204acf4-422d-441e-8128-2527e10c3e12 {"id":"9204acf4-422d-441e-8128-2527e10c3e12","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1445dd4 9204acf4-422d-441e-8128-2527e10c3e12 /tidb/cdc/default/default/upstream/7489360440687679498 {"id":7489360440687679498,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9204acf4-422d-441e-8128-2527e10c3e12 {"id":"9204acf4-422d-441e-8128-2527e10c3e12","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1445dd4 9204acf4-422d-441e-8128-2527e10c3e12 /tidb/cdc/default/default/upstream/7489360440687679498 {"id":7489360440687679498,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9204acf4-422d-441e-8128-2527e10c3e12 {"id":"9204acf4-422d-441e-8128-2527e10c3e12","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752622} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1445dd4 9204acf4-422d-441e-8128-2527e10c3e12 /tidb/cdc/default/default/upstream/7489360440687679498 {"id":7489360440687679498,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_stop_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-29536?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_stop_delay ++ date + echo '[Fri Apr 4 15:43:45 CST 2025] <<<<<< START storage consumer in processor_stop_delay case >>>>>>' [Fri Apr 4 15:43:45 CST 2025] <<<<<< START storage consumer in processor_stop_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_stop_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_stop_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-29536?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 8-th time, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360463922149768 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05e09c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:7033, start at 2025-04-04 15:43:44.168941195 +0800 CST m=+2.390788682 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:44.177 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:44.167 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:44.167 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360463922149768 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05e09c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:7033, start at 2025-04-04 15:43:44.168941195 +0800 CST m=+2.390788682 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:44.177 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:44.167 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:44.167 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360469369920642 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05e238000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:7099, start at 2025-04-04 15:43:44.300323892 +0800 CST m=+2.425886519 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:44.308 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:44.270 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:44.270 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_stop_delay.t not exists for 1-th check, retry later table split_region.test1 exists table split_region.test2 exists check diff successfully table batch_add_table.finish_mark not exists for 8-th check, retry later + set +x table changefeed_error.usertable not exists for 1-th check, retry later go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 table ddl_manager.finish_mark not exists for 17-th check, retry later go: downloading github.com/jmespath/go-jmespath v0.4.0 table processor_resolved_ts_fallback.t2 not exists for 3-th check, retry later check diff failed 9-th time, retry later check diff failed 5-th time, retry later check diff failed 3-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8408.out cli tso query --pd=http://127.0.0.1:2379 table processor_stop_delay.t not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:47 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e416740-c764-40e9-9800-5da0e7328003 {"id":"9e416740-c764-40e9-9800-5da0e7328003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14bd3d5 9e416740-c764-40e9-9800-5da0e7328003 /tidb/cdc/default/default/upstream/7489360454911571584 {"id":7489360454911571584,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e416740-c764-40e9-9800-5da0e7328003 {"id":"9e416740-c764-40e9-9800-5da0e7328003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14bd3d5 9e416740-c764-40e9-9800-5da0e7328003 /tidb/cdc/default/default/upstream/7489360454911571584 {"id":7489360454911571584,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e416740-c764-40e9-9800-5da0e7328003 {"id":"9e416740-c764-40e9-9800-5da0e7328003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc14bd3d5 9e416740-c764-40e9-9800-5da0e7328003 /tidb/cdc/default/default/upstream/7489360454911571584 {"id":7489360454911571584,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.cli.8014.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml Create changefeed successfully! ID: 668828bd-2269-467f-8d39-7a11f144ad5e Info: {"upstream_id":7489360454911571584,"namespace":"default","id":"668828bd-2269-467f-8d39-7a11f144ad5e","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:48.331643263+08:00","start_ts":457114288986783746,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288986783746,"checkpoint_ts":457114288986783746,"checkpoint_time":"2025-04-04 15:43:48.276"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table batch_add_table.finish_mark not exists for 9-th check, retry later table changefeed_error.usertable not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 18-th check, retry later table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 2670 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' check diff successfully Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later check diff successfully [Fri Apr 4 15:43:49 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.31643166.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/canal_json_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_basic ++ date + echo '[Fri Apr 4 15:43:49 CST 2025] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>>' [Fri Apr 4 15:43:49 CST 2025] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml + set +x + tso='457114289049960450 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114289049960450 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "mysql.host"="127.0.0.1" "workload"="core" "mysql.port"="4000" "operationcount"="0" "updateproportion"="0" "scanproportion"="0" "recordcount"="10" "mysql.user"="root" "readallfields"="true" "requestdistribution"="uniform" "mysql.db"="multi_capture_1" "dotransactions"="false" "threadcount"="2" "insertproportion"="0" "readproportion"="0" ********************************************** Run finished, takes 10.211915ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1569.9, Avg(us): 1968, Min(us): 1283, Max(us): 3799, 95th(us): 4000, 99th(us): 4000 table processor_stop_delay.t not exists for 3-th check, retry later ***************** properties ***************** "threadcount"="2" "mysql.port"="4000" "insertproportion"="0" "requestdistribution"="uniform" "dotransactions"="false" "readproportion"="0" "mysql.user"="root" "readallfields"="true" "updateproportion"="0" "scanproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="10" "operationcount"="0" "mysql.db"="multi_capture_2" "workload"="core" ********************************************** Run finished, takes 9.134053ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1834.1, Avg(us): 1765, Min(us): 1177, Max(us): 3604, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "insertproportion"="0" "mysql.user"="root" "recordcount"="10" "operationcount"="0" "workload"="core" "scanproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.db"="multi_capture_3" "readallfields"="true" "updateproportion"="0" "dotransactions"="false" "readproportion"="0" "mysql.port"="4000" "threadcount"="2" ********************************************** Run finished, takes 8.612758ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1923.3, Avg(us): 1636, Min(us): 1005, Max(us): 3349, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "readproportion"="0" "updateproportion"="0" "readallfields"="true" "mysql.user"="root" "recordcount"="10" "mysql.db"="multi_capture_4" "threadcount"="2" "requestdistribution"="uniform" "dotransactions"="false" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "workload"="core" "operationcount"="0" "scanproportion"="0" ********************************************** table batch_add_table.finish_mark not exists for 10-th check, retry later Run finished, takes 9.409513ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1844.0, Avg(us): 1788, Min(us): 1063, Max(us): 3904, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:43:50 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.85158517.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 19-th check, retry later table changefeed_error.usertable not exists for 3-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later table batch_add_table.finish_mark not exists for 11-th check, retry later check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... table processor_stop_delay.t not exists for 4-th check, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 20-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:52 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/ec138267-86e7-4420-a672-a9c58bddc753 {UpstreamID:7489360362934689504 Namespace:default ID:ec138267-86e7-4420-a672-a9c58bddc753 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:27.04954115 +0800 CST StartTs:457114283400757253 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283413864455} {CheckpointTs:457114289980571653 MinTableBarrierTs:457114289980571653 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/69c2781e-6503-47ba-ad34-2058fb90f1cf {"id":"69c2781e-6503-47ba-ad34-2058fb90f1cf","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752629} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f80134 69c2781e-6503-47ba-ad34-2058fb90f1cf /tidb/cdc/default/default/changefeed/info/ec138267-86e7-4420-a672-a9c58bddc753 {"upstream-id":7489360362934689504,"namespace":"default","changefeed-id":"ec138267-86e7-4420-a672-a9c58bddc753","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:27.04954115+08:00","start-ts":457114283400757253,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283413864455} /tidb/cdc/default/default/changefeed/status/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":457114289980571653,"min-table-barrier-ts":457114289980571653,"admin-job-type":0} /tidb/cdc/default/default/task/position/5af2726f-52b5-4a08-a605-5b110b7002ed/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/69c2781e-6503-47ba-ad34-2058fb90f1cf/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/ec138267-86e7-4420-a672-a9c58bddc753 {UpstreamID:7489360362934689504 Namespace:default ID:ec138267-86e7-4420-a672-a9c58bddc753 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:27.04954115 +0800 CST StartTs:457114283400757253 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283413864455} {CheckpointTs:457114289980571653 MinTableBarrierTs:457114289980571653 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/69c2781e-6503-47ba-ad34-2058fb90f1cf {"id":"69c2781e-6503-47ba-ad34-2058fb90f1cf","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752629} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f80134 69c2781e-6503-47ba-ad34-2058fb90f1cf /tidb/cdc/default/default/changefeed/info/ec138267-86e7-4420-a672-a9c58bddc753 {"upstream-id":7489360362934689504,"namespace":"default","changefeed-id":"ec138267-86e7-4420-a672-a9c58bddc753","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:27.04954115+08:00","start-ts":457114283400757253,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283413864455} /tidb/cdc/default/default/changefeed/status/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":457114289980571653,"min-table-barrier-ts":457114289980571653,"admin-job-type":0} /tidb/cdc/default/default/task/position/5af2726f-52b5-4a08-a605-5b110b7002ed/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/69c2781e-6503-47ba-ad34-2058fb90f1cf/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/ec138267-86e7-4420-a672-a9c58bddc753 {UpstreamID:7489360362934689504 Namespace:default ID:ec138267-86e7-4420-a672-a9c58bddc753 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:27.04954115 +0800 CST StartTs:457114283400757253 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde7e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114283413864455} {CheckpointTs:457114289980571653 MinTableBarrierTs:457114289980571653 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5af2726f-52b5-4a08-a605-5b110b7002ed {"id":"5af2726f-52b5-4a08-a605-5b110b7002ed","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752603} /tidb/cdc/default/__cdc_meta__/capture/69c2781e-6503-47ba-ad34-2058fb90f1cf {"id":"69c2781e-6503-47ba-ad34-2058fb90f1cf","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752629} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f7ffd7 5af2726f-52b5-4a08-a605-5b110b7002ed /tidb/cdc/default/__cdc_meta__/owner/223195ffc0f80134 69c2781e-6503-47ba-ad34-2058fb90f1cf /tidb/cdc/default/default/changefeed/info/ec138267-86e7-4420-a672-a9c58bddc753 {"upstream-id":7489360362934689504,"namespace":"default","changefeed-id":"ec138267-86e7-4420-a672-a9c58bddc753","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-22200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:27.04954115+08:00","start-ts":457114283400757253,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114283413864455} /tidb/cdc/default/default/changefeed/status/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":457114289980571653,"min-table-barrier-ts":457114289980571653,"admin-job-type":0} /tidb/cdc/default/default/task/position/5af2726f-52b5-4a08-a605-5b110b7002ed/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/69c2781e-6503-47ba-ad34-2058fb90f1cf/ec138267-86e7-4420-a672-a9c58bddc753 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360362934689504 {"id":7489360362934689504,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:53 CST 2025] <<<<<< run test case capture_session_done_during_task success! >>>>>> table changefeed_error.usertable not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:53 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:53 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.85758577.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_stop_delay.t not exists for 5-th check, retry later table batch_add_table.finish_mark not exists for 12-th check, retry later check diff failed 3-th time, retry later table changefeed_error.usertable exists check diff successfully ***************** properties ***************** "updateproportion"="0" "mysql.user"="root" "mysql.host"="127.0.0.1" "dotransactions"="false" "scanproportion"="0" "workload"="core" "requestdistribution"="uniform" "readproportion"="0" "threadcount"="4" "mysql.port"="4000" "operationcount"="0" "mysql.db"="changefeed_error" "insertproportion"="0" "recordcount"="20" "readallfields"="true" ********************************************** Run finished, takes 6.222995ms INSERT - Takes(s): 0.0, Count: 19, OPS: 6098.0, Avg(us): 1041, Min(us): 481, Max(us): 3068, 95th(us): 4000, 99th(us): 4000 check diff successfully {"id":"3cd2a78f-f33f-4cf6-b3cb-a51082e478b6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752620} table ddl_manager.finish_mark not exists for 21-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + [[ /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 1-th time, retry later table processor_stop_delay.t exists check diff successfully start tidb cluster in /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table batch_add_table.finish_mark not exists for 13-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:56 GMT < Content-Length: 1277 < Content-Type: text/plain; charset=utf-8 < { [1277 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:56 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.86288630.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8303 failed: Connection refused * Failed to connect to 127.0.0.1 port 8303: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + [[ /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later check diff failed 1-th time, retry later check diff failed 2-th time, retry later check diff failed 1-th time, retry later table batch_add_table.finish_mark not exists for 14-th check, retry later check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 23-th check, retry later Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8303 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:59 GMT < Content-Length: 1759 < Content-Type: text/plain; charset=utf-8 < { [1759 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 {"id":"1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752636} /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa47e 1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 {"id":"1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752636} /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa47e 1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 {"id":"1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752636} /tidb/cdc/default/__cdc_meta__/capture/747b1128-fb5f-4540-a8a9-4731c3c34a1a {"id":"747b1128-fb5f-4540-a8a9-4731c3c34a1a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752633} /tidb/cdc/default/__cdc_meta__/capture/ca6f2691-e09e-4bc7-b22d-84364937b0ea {"id":"ca6f2691-e09e-4bc7-b22d-84364937b0ea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752630} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa427 ca6f2691-e09e-4bc7-b22d-84364937b0ea /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa45d 747b1128-fb5f-4540-a8a9-4731c3c34a1a /tidb/cdc/default/__cdc_meta__/owner/223195ffc15aa47e 1761fec8-8cd2-485b-81c0-fa8e5b7ed5d2 /tidb/cdc/default/default/upstream/7489360463922149768 {"id":7489360463922149768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8681.out cli changefeed create --start-ts=457114289049960450 '--sink-uri=file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-19079?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: 529dd321-98bc-4fe9-ae5c-4781e7332add Info: {"upstream_id":7489360463922149768,"namespace":"default","id":"529dd321-98bc-4fe9-ae5c-4781e7332add","sink_uri":"file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-19079?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:59.949925665+08:00","start_ts":457114289049960450,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114289049960450,"checkpoint_ts":457114289049960450,"checkpoint_time":"2025-04-04 15:43:48.517"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later table batch_add_table.finish_mark not exists for 15-th check, retry later check diff failed 3-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... table ddl_manager.finish_mark not exists for 24-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/multi_capture + sink_uri='file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-19079?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_capture ++ date + echo '[Fri Apr 4 15:44:01 CST 2025] <<<<<< START storage consumer in multi_capture case >>>>>>' [Fri Apr 4 15:44:01 CST 2025] <<<<<< START storage consumer in multi_capture case >>>>>> + cd /tmp/tidb_cdc_test/multi_capture + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_capture + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_capture/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-19079?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table multi_capture_1.usertable not exists for 1-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + [[ /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/3cd2a78f-f33f-4cf6-b3cb-a51082e478b6 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later check diff failed 6-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully [Fri Apr 4 15:44:02 CST 2025] <<<<<< run test case canal_json_storage_basic success! >>>>>> table batch_add_table.finish_mark not exists for 16-th check, retry later check diff failed 4-th time, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 25-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table multi_capture_1.usertable not exists for 2-th check, retry later check diff failed 7-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/csv_storage_update_pk_clustered Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 5-th time, retry later check diff failed 2-th time, retry later check diff failed 4-th time, retry later table batch_add_table.finish_mark not exists for 17-th check, retry later Starting Upstream TiDB... check diff successfully table ddl_manager.finish_mark not exists for 26-th check, retry later check diff failed 8-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360548520059507 TiDB Cluster ID. tikv_gc_leader_uuid 657ff070bec0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:7807, start at 2025-04-04 15:44:03.331847611 +0800 CST m=+2.267928526 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:03.340 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:03.323 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:03.323 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360548520059507 TiDB Cluster ID. tikv_gc_leader_uuid 657ff070bec0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:7807, start at 2025-04-04 15:44:03.331847611 +0800 CST m=+2.267928526 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:03.340 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:03.323 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:03.323 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360555756100139 TiDB Cluster ID. tikv_gc_leader_uuid 657ff070c40000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:7882, start at 2025-04-04 15:44:03.369828279 +0800 CST m=+2.222589274 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:03.377 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:03.423 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:03.423 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_1.usertable not exists for 3-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark not exists for 18-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:06 CST 2025] <<<<<< run test case split_region success! >>>>>> check diff failed 6-th time, retry later check diff failed 3-th time, retry later check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 27-th check, retry later check diff failed 9-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360556288501325 TiDB Cluster ID. tikv_gc_leader_uuid 657ff072be4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:3880, start at 2025-04-04 15:44:05.397118473 +0800 CST m=+4.175651464 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:05.404 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:05.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:05.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360556288501325 TiDB Cluster ID. tikv_gc_leader_uuid 657ff072be4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:3880, start at 2025-04-04 15:44:05.397118473 +0800 CST m=+4.175651464 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:05.404 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:05.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:05.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360551171970415 TiDB Cluster ID. tikv_gc_leader_uuid 657ff070d94000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-mrm-0mhsv, pid:3954, start at 2025-04-04 15:44:03.457701315 +0800 CST m=+2.150592665 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:03.464 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:03.482 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:03.482 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table multi_capture_1.usertable not exists for 4-th check, retry later [Fri Apr 4 15:44:07 CST 2025] <<<<<< START cdc server in csv_storage_multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_multi_tables_ddl.92199221.out server --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:44:07 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.58655867.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 19-th check, retry later check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 check diff successfully start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 6-th time, retry later table ddl_manager.finish_mark not exists for 28-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... wait process cdc.test exit for 1-th time... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table multi_capture_1.usertable not exists for 5-th check, retry later check diff successfully [Fri Apr 4 15:44:09 CST 2025] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.53395341.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:10 CST 2025] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360567750160992 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0758b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:3973, start at 2025-04-04 15:44:08.264199426 +0800 CST m=+2.347408487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:08.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:08.238 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:08.238 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360567750160992 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0758b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:3973, start at 2025-04-04 15:44:08.264199426 +0800 CST m=+2.347408487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:08.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:08.238 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:08.238 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360558716103157 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07599c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:4048, start at 2025-04-04 15:44:08.313249139 +0800 CST m=+2.291794507 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:08.321 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:08.295 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:08.295 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table batch_add_table.finish_mark not exists for 20-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:10 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5c4f3ac5-c962-47fe-8e05-8b23f7a23496 {"id":"5c4f3ac5-c962-47fe-8e05-8b23f7a23496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a81ad9 5c4f3ac5-c962-47fe-8e05-8b23f7a23496 /tidb/cdc/default/default/upstream/7489360548520059507 {"id":7489360548520059507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5c4f3ac5-c962-47fe-8e05-8b23f7a23496 {"id":"5c4f3ac5-c962-47fe-8e05-8b23f7a23496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a81ad9 5c4f3ac5-c962-47fe-8e05-8b23f7a23496 /tidb/cdc/default/default/upstream/7489360548520059507 {"id":7489360548520059507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5c4f3ac5-c962-47fe-8e05-8b23f7a23496 {"id":"5c4f3ac5-c962-47fe-8e05-8b23f7a23496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a81ad9 5c4f3ac5-c962-47fe-8e05-8b23f7a23496 /tidb/cdc/default/default/upstream/7489360548520059507 {"id":7489360548520059507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:10 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3433 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5cab7581-b27f-4b0e-9493-5247e24a6496 {"id":"5cab7581-b27f-4b0e-9493-5247e24a6496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373017 5cab7581-b27f-4b0e-9493-5247e24a6496 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360427031806945,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:43.480953984+08:00","start-ts":457114286408073217,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:44:08.199892897+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114294206857218} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114290418876425,"min-table-barrier-ts":457114290418876425,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5cab7581-b27f-4b0e-9493-5247e24a6496 {"id":"5cab7581-b27f-4b0e-9493-5247e24a6496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373017 5cab7581-b27f-4b0e-9493-5247e24a6496 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360427031806945,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:43.480953984+08:00","start-ts":457114286408073217,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:44:08.199892897+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114294206857218} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114290418876425,"min-table-barrier-ts":457114290418876425,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5cab7581-b27f-4b0e-9493-5247e24a6496 {"id":"5cab7581-b27f-4b0e-9493-5247e24a6496","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752647} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373017 5cab7581-b27f-4b0e-9493-5247e24a6496 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489360427031806945,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:43.480953984+08:00","start-ts":457114286408073217,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:44:08.199892897+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114294206857218} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114290418876425,"min-table-barrier-ts":457114290418876425,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7489360548520059507,"namespace":"default","id":"test-normal","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-16725?flush-interval=5s\u0026protocol=csv","create_time":"2025-04-04T15:44:10.818196111+08:00","start_ts":457114294038560770,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114294038560770,"checkpoint_ts":457114294038560770,"checkpoint_time":"2025-04-04 15:44:07.547"} + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114290418876425, "checkpoint_time": "2025-04-04 15:43:53.739", "error": { "time": "2025-04-04T15:44:08.199892897+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114290418876425, "checkpoint_time": "2025-04-04 15:43:53.739", "error": { "time": "2025-04-04T15:44:08.199892897+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114290418876425, "checkpoint_time": "2025-04-04 15:43:53.739", "error": { "time": "2025-04-04T15:44:08.199892897+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114290418876425, '"checkpoint_time":' '"2025-04-04' '15:43:53.739",' '"error":' '{' '"time":' '"2025-04-04T15:44:08.199892897+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114290418876425, '"checkpoint_time":' '"2025-04-04' '15:43:53.739",' '"error":' '{' '"time":' '"2025-04-04T15:44:08.199892897+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5958.out cli changefeed remove -c changefeed-error table ddl_manager.finish_mark not exists for 29-th check, retry later Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7489360548520059507,"namespace":"default","id":"test-error-1","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-13474?flush-interval=5s\u0026protocol=csv","create_time":"2025-04-04T15:44:10.967484902+08:00","start_ts":457114294038560770,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114294038560770,"checkpoint_ts":457114294038560770,"checkpoint_time":"2025-04-04 15:44:07.547"} Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7489360548520059507,"namespace":"default","id":"test-error-2","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-10861?flush-interval=5s\u0026protocol=csv","create_time":"2025-04-04T15:44:11.116466579+08:00","start_ts":457114294038560770,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114294038560770,"checkpoint_ts":457114294038560770,"checkpoint_time":"2025-04-04 15:44:07.547"} + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-16725?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + log_suffix=1 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer1.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-16725?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-13474?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + log_suffix=2 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer2.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-13474?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-10861?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml + log_suffix=3 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 4 15:44:11 CST 2025] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer3.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-10861?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later table multi_capture_1.usertable not exists for 6-th check, retry later Changefeed remove successfully. ID: changefeed-error CheckpointTs: 457114290418876425 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table multi_tables_ddl_test.t55 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5288.out cli tso query --pd=http://127.0.0.1:2379 table batch_add_table.finish_mark not exists for 21-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:12 GMT < Content-Length: 862 < Content-Type: text/plain; charset=utf-8 < { [862 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f871b104-2015-48eb-a814-3c4ec109e84e {"id":"f871b104-2015-48eb-a814-3c4ec109e84e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a6a4e5 f871b104-2015-48eb-a814-3c4ec109e84e /tidb/cdc/default/default/upstream/7489360556288501325 {"id":7489360556288501325,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f871b104-2015-48eb-a814-3c4ec109e84e {"id":"f871b104-2015-48eb-a814-3c4ec109e84e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a6a4e5 f871b104-2015-48eb-a814-3c4ec109e84e /tidb/cdc/default/default/upstream/7489360556288501325 {"id":7489360556288501325,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f871b104-2015-48eb-a814-3c4ec109e84e {"id":"f871b104-2015-48eb-a814-3c4ec109e84e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1a6a4e5 f871b104-2015-48eb-a814-3c4ec109e84e /tidb/cdc/default/default/upstream/7489360556288501325 {"id":7489360556288501325,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: ba3fdfea-6b5b-4478-8a8a-b03553cc03d6 Info: {"upstream_id":7489360556288501325,"namespace":"default","id":"ba3fdfea-6b5b-4478-8a8a-b03553cc03d6","sink_uri":"file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-3099?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:12.828087481+08:00","start_ts":457114295394369543,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114295394369543,"checkpoint_ts":457114295394369543,"checkpoint_time":"2025-04-04 15:44:12.719"} + workdir=/tmp/tidb_cdc_test/region_merge + sink_uri='file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-3099?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/region_merge ++ date + echo '[Fri Apr 4 15:44:12 CST 2025] <<<<<< START storage consumer in region_merge case >>>>>>' [Fri Apr 4 15:44:12 CST 2025] <<<<<< START storage consumer in region_merge case >>>>>> + cd /tmp/tidb_cdc_test/region_merge + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/region_merge + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/region_merge/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-3099?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 30-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release split_and_random_merge scale: 20 *************************** 1. row *************************** count(distinct region_id): 1 + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully check diff failed 8-th time, retry later table multi_capture_1.usertable exists table multi_capture_2.usertable exists table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360589440889998 TiDB Cluster ID. tikv_gc_leader_uuid 657ff079bd00033 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:7871, start at 2025-04-04 15:44:12.558839341 +0800 CST m=+3.339008068 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:12.569 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:12.532 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:12.532 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360589440889998 TiDB Cluster ID. tikv_gc_leader_uuid 657ff079bd00033 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:7871, start at 2025-04-04 15:44:12.558839341 +0800 CST m=+3.339008068 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:12.569 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:12.532 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:12.532 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360579185751575 TiDB Cluster ID. tikv_gc_leader_uuid 657ff079be0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:7943, start at 2025-04-04 15:44:12.565081154 +0800 CST m=+3.230658032 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:12.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:12.536 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:12.536 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_clustered/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_clustered/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_clustered/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114295385980929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114295385980929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Fri Apr 4 15:44:14 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.53255327.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' wait process cdc.test exit for 2-th time... Starting Upstream TiDB... + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' table multi_tables_ddl_test.t55 not exists for 2-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:44:14 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60496051.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 22-th check, retry later check diff successfully table ddl_manager.finish_mark not exists for 31-th check, retry later [Fri Apr 4 15:44:15 CST 2025] <<<<<< START cdc server in csv_storage_update_pk_clustered case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.91629164.out server --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... table multi_capture_4.usertable not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... table multi_tables_ddl_test.t55 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark exists check diff successfully wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 32-th check, retry later check diff failed 4-th time, retry later wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:44:17 CST 2025] <<<<<< run test case processor_stop_delay success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:17 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3724b4f9-a5e8-482b-b4a8-564edb297f90 {"id":"3724b4f9-a5e8-482b-b4a8-564edb297f90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1b332f1 3724b4f9-a5e8-482b-b4a8-564edb297f90 /tidb/cdc/default/default/upstream/7489360567750160992 {"id":7489360567750160992,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3724b4f9-a5e8-482b-b4a8-564edb297f90 {"id":"3724b4f9-a5e8-482b-b4a8-564edb297f90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1b332f1 3724b4f9-a5e8-482b-b4a8-564edb297f90 /tidb/cdc/default/default/upstream/7489360567750160992 {"id":7489360567750160992,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3724b4f9-a5e8-482b-b4a8-564edb297f90 {"id":"3724b4f9-a5e8-482b-b4a8-564edb297f90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1b332f1 3724b4f9-a5e8-482b-b4a8-564edb297f90 /tidb/cdc/default/default/upstream/7489360567750160992 {"id":7489360567750160992,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 2-th time... table multi_capture_4.usertable not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:17 GMT < Content-Length: 843 < Content-Type: text/plain; charset=utf-8 < { [843 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c525a430-3ab5-4eb2-b097-1e79d3217ed8 {"id":"c525a430-3ab5-4eb2-b097-1e79d3217ed8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373072 c525a430-3ab5-4eb2-b097-1e79d3217ed8 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c525a430-3ab5-4eb2-b097-1e79d3217ed8 {"id":"c525a430-3ab5-4eb2-b097-1e79d3217ed8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373072 c525a430-3ab5-4eb2-b097-1e79d3217ed8 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c525a430-3ab5-4eb2-b097-1e79d3217ed8 {"id":"c525a430-3ab5-4eb2-b097-1e79d3217ed8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752654} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373072 c525a430-3ab5-4eb2-b097-1e79d3217ed8 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6103.out cli changefeed create --start-ts=457114286408073217 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7489360427031806945,"namespace":"default","id":"changefeed-error-1","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:18.123915298+08:00","start_ts":457114286408073217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217,"checkpoint_time":"2025-04-04 15:43:38.439"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table multi_tables_ddl_test.t55 not exists for 4-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit \033[0;36m<<< Run all test success >>>\033[0m + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:18 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40444d0b-01e4-46eb-8d5e-544badcb41ab {"id":"40444d0b-01e4-46eb-8d5e-544badcb41ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c5e9d8 40444d0b-01e4-46eb-8d5e-544badcb41ab /tidb/cdc/default/default/upstream/7489360589440889998 {"id":7489360589440889998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40444d0b-01e4-46eb-8d5e-544badcb41ab {"id":"40444d0b-01e4-46eb-8d5e-544badcb41ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c5e9d8 40444d0b-01e4-46eb-8d5e-544badcb41ab /tidb/cdc/default/default/upstream/7489360589440889998 {"id":7489360589440889998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/40444d0b-01e4-46eb-8d5e-544badcb41ab {"id":"40444d0b-01e4-46eb-8d5e-544badcb41ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752656} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1c5e9d8 40444d0b-01e4-46eb-8d5e-544badcb41ab /tidb/cdc/default/default/upstream/7489360589440889998 {"id":7489360589440889998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.cli.9219.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache table ddl_manager.finish_mark not exists for 33-th check, retry later check diff failed 5-th time, retry later [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage *************************** 1. row *************************** count(distinct region_id): 20 [Pipeline] } + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1343 0 --:--:-- --:--:-- --:--:-- 1343 + info='{"state":"normal","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217}' + echo '{"state":"normal","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217}' {"state":"normal","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217} ++ echo '{"state":"normal","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217}' ++ jq -r .state + state=normal + [[ ! normal == \w\a\r\n\i\n\g ]] + echo 'changefeed state normal does not equal to warning' changefeed state normal does not equal to warning + exit 1 run task failed 1-th time, retry later [Pipeline] // container start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] } table multi_capture_4.usertable not exists for 4-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table multi_tables_ddl_test.t55 not exists for 5-th check, retry later [Pipeline] // stage [Pipeline] } + set +x + tso='457114297101975559 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114297101975559 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:20 CST 2025] <<<<<< run test case owner_resign success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360612144146822 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07fff40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:4185, start at 2025-04-04 15:44:18.956625169 +0800 CST m=+4.417285941 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:19.067 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:19.142 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:19.142 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360612144146822 TiDB Cluster ID. tikv_gc_leader_uuid 657ff07fff40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:4185, start at 2025-04-04 15:44:18.956625169 +0800 CST m=+4.417285941 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:19.067 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:19.142 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:19.142 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360610508698781 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0803f0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-tvh-44ssr, pid:4258, start at 2025-04-04 15:44:19.240134251 +0800 CST m=+4.628084520 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:19.247 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:19.247 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:19.247 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 34-th check, retry later check diff failed 6-th time, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table multi_capture_4.usertable not exists for 5-th check, retry later table multi_tables_ddl_test.t55 not exists for 6-th check, retry later check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 3123 0 --:--:-- --:--:-- --:--:-- 3123 + info='{"state":"warning","resolved_ts":457114286788182035,"checkpoint_ts":457114286788182035,"last_warning":{"time":"2025-04-04T15:44:20.014878507+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":457114286788182035,"checkpoint_ts":457114286788182035,"last_warning":{"time":"2025-04-04T15:44:20.014878507+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":457114286788182035,"checkpoint_ts":457114286788182035,"last_warning":{"time":"2025-04-04T15:44:20.014878507+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":457114286788182035,"checkpoint_ts":457114286788182035,"last_warning":{"time":"2025-04-04T15:44:20.014878507+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":457114286788182035,"checkpoint_ts":457114286788182035,"last_warning":{"time":"2025-04-04T15:44:20.014878507+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6149.out cli changefeed remove -c changefeed-error-1 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 457114286788182035 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] // stage [Pipeline] } [Pipeline] // container table ddl_manager.finish_mark not exists for 35-th check, retry later [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.cli.9262.out cli changefeed create --start-ts=457114297101975559 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed1?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed1.toml -c changefeed1 Create changefeed successfully! ID: changefeed1 Info: {"upstream_id":7489360589440889998,"namespace":"default","id":"changefeed1","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed1?flush-interval=5s","create_time":"2025-04-04T15:44:23.357914211+08:00","start_ts":457114297101975559,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":false},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297101975559,"checkpoint_ts":457114297101975559,"checkpoint_time":"2025-04-04 15:44:19.233"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // withEnv [Pipeline] } table multi_capture_4.usertable exists [Pipeline] // stage check diff failed 7-th time, retry later [Pipeline] } split_and_random_merge scale: 40 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5559.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully ***************** properties ***************** "readallfields"="true" "mysql.user"="root" "workload"="core" "mysql.db"="multi_capture_1" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "operationcount"="0" "updateproportion"="0" "insertproportion"="0" "readproportion"="0" "mysql.port"="4000" "dotransactions"="false" "recordcount"="20" "scanproportion"="0" "threadcount"="2" ********************************************** Run finished, takes 15.393798ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1502.5, Avg(us): 1494, Min(us): 459, Max(us): 5161, 95th(us): 5000, 99th(us): 6000 ***************** properties ***************** "mysql.user"="root" "mysql.db"="multi_capture_2" "workload"="core" "updateproportion"="0" "dotransactions"="false" "operationcount"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "recordcount"="20" "insertproportion"="0" "requestdistribution"="uniform" "readproportion"="0" "threadcount"="2" "mysql.port"="4000" "scanproportion"="0" ********************************************** Run finished, takes 13.035972ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1701.3, Avg(us): 1196, Min(us): 501, Max(us): 2389, 95th(us): 2000, 99th(us): 3000 ***************** properties ***************** "insertproportion"="0" "readproportion"="0" "mysql.user"="root" "operationcount"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "readallfields"="true" "requestdistribution"="uniform" "threadcount"="2" "recordcount"="20" "workload"="core" "scanproportion"="0" "mysql.port"="4000" "updateproportion"="0" "mysql.db"="multi_capture_3" ********************************************** Run finished, takes 11.662464ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1901.7, Avg(us): 1122, Min(us): 503, Max(us): 1841, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "threadcount"="2" "operationcount"="0" "workload"="core" "readproportion"="0" "requestdistribution"="uniform" "scanproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "updateproportion"="0" "recordcount"="20" "mysql.user"="root" "mysql.db"="multi_capture_4" "mysql.port"="4000" "readallfields"="true" ********************************************** Run finished, takes 11.698829ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2014.8, Avg(us): 1124, Min(us): 503, Max(us): 1956, 95th(us): 2000, 99th(us): 2000 *************************** 1. row *************************** count(distinct region_id): 4 check diff failed 1-th time, retry later + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... + set +x + tso='457114298257768449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114298257768449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:25 CST 2025] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.55895591.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed1?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed1.toml + log_suffix=changefeed1 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered ++ date + echo '[Fri Apr 4 15:44:24 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>>' [Fri Apr 4 15:44:24 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed1.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc_storage_consumerchangefeed1.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed1?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed1.toml table ddl_manager.finish_mark not exists for 36-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... check diff failed 8-th time, retry later process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:44:25 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.62066208.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_tables_ddl_test.t66 not exists for 2-th check, retry later check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 37-th check, retry later check diff failed 9-th time, retry later table multi_tables_ddl_test.t66 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:28 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bae6f2f-160f-4c9f-86b4-a12cb85745a9 {"id":"4bae6f2f-160f-4c9f-86b4-a12cb85745a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d8b4ea 4bae6f2f-160f-4c9f-86b4-a12cb85745a9 /tidb/cdc/default/default/upstream/7489360612144146822 {"id":7489360612144146822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bae6f2f-160f-4c9f-86b4-a12cb85745a9 {"id":"4bae6f2f-160f-4c9f-86b4-a12cb85745a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d8b4ea 4bae6f2f-160f-4c9f-86b4-a12cb85745a9 /tidb/cdc/default/default/upstream/7489360612144146822 {"id":7489360612144146822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bae6f2f-160f-4c9f-86b4-a12cb85745a9 {"id":"4bae6f2f-160f-4c9f-86b4-a12cb85745a9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1d8b4ea 4bae6f2f-160f-4c9f-86b4-a12cb85745a9 /tidb/cdc/default/default/upstream/7489360612144146822 {"id":7489360612144146822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5640.out cli changefeed create --start-ts=457114298257768449 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-31932?protocol=canal-json&enable-tidb-extension=true' check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360649042763877 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0873c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:9224, start at 2025-04-04 15:44:26.398104847 +0800 CST m=+2.369696621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:26.407 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:26.403 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:26.403 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360649042763877 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0873c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:9224, start at 2025-04-04 15:44:26.398104847 +0800 CST m=+2.369696621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:26.407 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:26.403 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:26.403 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360647059704459 TiDB Cluster ID. tikv_gc_leader_uuid 657ff087470000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-bd9-jqhdr, pid:9290, start at 2025-04-04 15:44:26.420540651 +0800 CST m=+2.329264992 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:26.467 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Create changefeed successfully! ID: 528f01c1-1eb5-4501-a1e1-cdfa0e33aa1c Info: {"upstream_id":7489360612144146822,"namespace":"default","id":"528f01c1-1eb5-4501-a1e1-cdfa0e33aa1c","sink_uri":"file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-31932?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:28.631430536+08:00","start_ts":457114298257768449,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114298257768449,"checkpoint_ts":457114298257768449,"checkpoint_time":"2025-04-04 15:44:23.642"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:28 GMT < Content-Length: 843 < Content-Type: text/plain; charset=utf-8 < { [843 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed3c9cad-d404-49b5-abc1-0edf852bc68d {"id":"ed3c9cad-d404-49b5-abc1-0edf852bc68d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13730f3 ed3c9cad-d404-49b5-abc1-0edf852bc68d /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed3c9cad-d404-49b5-abc1-0edf852bc68d {"id":"ed3c9cad-d404-49b5-abc1-0edf852bc68d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13730f3 ed3c9cad-d404-49b5-abc1-0edf852bc68d /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed3c9cad-d404-49b5-abc1-0edf852bc68d {"id":"ed3c9cad-d404-49b5-abc1-0edf852bc68d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752665} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13730f3 ed3c9cad-d404-49b5-abc1-0edf852bc68d /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6259.out cli changefeed create --start-ts=457114286408073217 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 table ddl_manager.finish_mark not exists for 38-th check, retry later Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7489360427031806945,"namespace":"default","id":"changefeed-error-2","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:29.051557392+08:00","start_ts":457114286408073217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114286408073217,"checkpoint_ts":457114286408073217,"checkpoint_time":"2025-04-04 15:43:38.439"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff successfully + set +x + workdir=/tmp/tidb_cdc_test/ddl_attributes + sink_uri='file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-31932?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_attributes ++ date + echo '[Fri Apr 4 15:44:30 CST 2025] <<<<<< START storage consumer in ddl_attributes case >>>>>>' [Fri Apr 4 15:44:30 CST 2025] <<<<<< START storage consumer in ddl_attributes case >>>>>> + cd /tmp/tidb_cdc_test/ddl_attributes + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_attributes + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-31932?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table multi_tables_ddl_test.t66 not exists for 4-th check, retry later check diff failed 4-th time, retry later [Fri Apr 4 15:44:30 CST 2025] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> *************************** 1. row *************************** count(distinct region_id): 40 shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1054310545.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc0 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:44:29.143568999+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:44:29.143568999+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114286408073217, "checkpoint_time": "2025-04-04 15:43:38.439", "error": { "time": "2025-04-04T15:44:29.143568999+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114286408073217, '"checkpoint_time":' '"2025-04-04' '15:43:38.439",' '"error":' '{' '"time":' '"2025-04-04T15:44:29.143568999+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114286408073217, '"checkpoint_time":' '"2025-04-04' '15:43:38.439",' '"error":' '{' '"time":' '"2025-04-04T15:44:29.143568999+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114286408073217 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6328.out cli changefeed remove -c changefeed-error-2 table ddl_manager.finish_mark not exists for 39-th check, retry later Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 457114286408073217 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 2-th time, retry later check diff successfully table ddl_manager.finish_mark not exists for 40-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... + set +x check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:33 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:44:33 CST 2025] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1059910601.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:34 CST 2025] <<<<<< run test case multi_capture success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:34 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.63796381.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_tables_ddl_test.finish_mark not exists for 2-th check, retry later Verifying downstream PD is started... check diff failed 2-th time, retry later check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 41-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... split_and_random_merge scale: 80 *************************** 1. row *************************** count(distinct region_id): 10 table multi_tables_ddl_test.finish_mark not exists for 3-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:36 GMT < Content-Length: 1277 < Content-Type: text/plain; charset=utf-8 < { [1277 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:44:36 CST 2025] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1065410656.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.cli.9458.out cli changefeed create --start-ts=457114297101975559 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed2?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed2.toml -c changefeed2 table ddl_manager.finish_mark not exists for 42-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:37 GMT < Content-Length: 843 < Content-Type: text/plain; charset=utf-8 < { [843 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afebfe48-4a28-4816-b744-889e4265f294 {"id":"afebfe48-4a28-4816-b744-889e4265f294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752674} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373178 afebfe48-4a28-4816-b744-889e4265f294 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afebfe48-4a28-4816-b744-889e4265f294 {"id":"afebfe48-4a28-4816-b744-889e4265f294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752674} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373178 afebfe48-4a28-4816-b744-889e4265f294 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afebfe48-4a28-4816-b744-889e4265f294 {"id":"afebfe48-4a28-4816-b744-889e4265f294","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752674} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1373178 afebfe48-4a28-4816-b744-889e4265f294 /tidb/cdc/default/default/upstream/7489360427031806945 {"id":7489360427031806945,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6434.out cli changefeed create --start-ts=0 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7489360427031806945,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:37.88946587+08:00","start_ts":457114301966319625,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114301966319625,"checkpoint_ts":457114301966319625,"checkpoint_time":"2025-04-04 15:44:37.789"} PASS Create changefeed successfully! ID: changefeed2 Info: {"upstream_id":7489360589440889998,"namespace":"default","id":"changefeed2","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed2?flush-interval=5s","create_time":"2025-04-04T15:44:37.657422056+08:00","start_ts":457114297101975559,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":true,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":true},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297101975559,"checkpoint_ts":457114297101975559,"checkpoint_time":"2025-04-04 15:44:19.233"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... coverage: 2.5% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table multi_tables_ddl_test.finish_mark not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 43-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed2?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed2.toml + log_suffix=changefeed2 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered ++ date + echo '[Fri Apr 4 15:44:39 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>>' [Fri Apr 4 15:44:39 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed2.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc_storage_consumerchangefeed2.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed2?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed2.toml + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 4 15:44:37 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ true != \n\o ]] + set +x + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.54215423.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 5-th time, retry later + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114301966319625, '"checkpoint_time":' '"2025-04-04' '15:44:37.789",' '"error":' '{' '"time":' '"2025-04-04T15:44:38.0814906+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:39 GMT < Content-Length: 1759 < Content-Type: text/plain; charset=utf-8 < { [1759 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/746fb900-396c-4db7-814d-2c64c0ef23ce {"id":"746fb900-396c-4db7-814d-2c64c0ef23ce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c536 746fb900-396c-4db7-814d-2c64c0ef23ce /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/746fb900-396c-4db7-814d-2c64c0ef23ce {"id":"746fb900-396c-4db7-814d-2c64c0ef23ce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c536 746fb900-396c-4db7-814d-2c64c0ef23ce /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 {"id":"2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752673} /tidb/cdc/default/__cdc_meta__/capture/746fb900-396c-4db7-814d-2c64c0ef23ce {"id":"746fb900-396c-4db7-814d-2c64c0ef23ce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752677} /tidb/cdc/default/__cdc_meta__/capture/eec74734-fdf5-416a-ad88-ab5ec9b117e2 {"id":"eec74734-fdf5-416a-ad88-ab5ec9b117e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752670} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c4d6 eec74734-fdf5-416a-ad88-ab5ec9b117e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c516 2acb0d84-4c74-40aa-8f78-5a1a3d45f3e4 /tidb/cdc/default/__cdc_meta__/owner/223195ffc201c536 746fb900-396c-4db7-814d-2c64c0ef23ce /tidb/cdc/default/default/upstream/7489360649042763877 {"id":7489360649042763877,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.cli.10712.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml sink.enable-partition-separator is deprecated and will be removed in future versions. The default value for sink.enable-partition-separator is `true`. Changing it to `false` may lead to data inconsistency issues. Please update your configuration as needed. Create changefeed successfully! ID: 41d0f0b6-31ce-4deb-9216-eef18382d5f3 Info: {"upstream_id":7489360649042763877,"namespace":"default","id":"41d0f0b6-31ce-4deb-9216-eef18382d5f3","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:40.105112446+08:00","start_ts":457114302559813636,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114302559813636,"checkpoint_ts":457114302559813636,"checkpoint_time":"2025-04-04 15:44:40.053"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table multi_tables_ddl_test.finish_mark not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 44-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 6-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114301966319625, "checkpoint_time": "2025-04-04 15:44:37.789", "error": { "time": "2025-04-04T15:44:38.0814906+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114301966319625, '"checkpoint_time":' '"2025-04-04' '15:44:37.789",' '"error":' '{' '"time":' '"2025-04-04T15:44:38.0814906+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later table multi_tables_ddl_test.finish_mark exists check table exists success Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) *************************** 1. row *************************** count(distinct region_id): 80 + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s + info='{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114302846861321, "checkpoint_time": "2025-04-04 15:44:41.148", "error": null }' + echo '{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114302846861321, "checkpoint_time": "2025-04-04 15:44:41.148", "error": null }' { "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114302846861321, "checkpoint_time": "2025-04-04 15:44:41.148", "error": null } ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114302846861321, '"checkpoint_time":' '"2025-04-04' '15:44:41.148",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114302846861321, '"checkpoint_time":' '"2025-04-04' '15:44:41.148",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s + info='{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114303108743173, "checkpoint_time": "2025-04-04 15:44:42.147", "error": null }' + echo '{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114303108743173, "checkpoint_time": "2025-04-04 15:44:42.147", "error": null }' { "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114303108743173, "checkpoint_time": "2025-04-04 15:44:42.147", "error": null } ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114303108743173, '"checkpoint_time":' '"2025-04-04' '15:44:42.147",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114303108743173, '"checkpoint_time":' '"2025-04-04' '15:44:42.147",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + info='{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114294038560770, "checkpoint_time": "2025-04-04 15:44:07.547", "error": { "time": "2025-04-04T15:44:12.293377504+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114294038560770, "checkpoint_time": "2025-04-04 15:44:07.547", "error": { "time": "2025-04-04T15:44:12.293377504+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7489360548520059507, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114294038560770, "checkpoint_time": "2025-04-04 15:44:07.547", "error": { "time": "2025-04-04T15:44:12.293377504+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114294038560770, '"checkpoint_time":' '"2025-04-04' '15:44:07.547",' '"error":' '{' '"time":' '"2025-04-04T15:44:12.293377504+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489360548520059507, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114294038560770, '"checkpoint_time":' '"2025-04-04' '15:44:07.547",' '"error":' '{' '"time":' '"2025-04-04T15:44:12.293377504+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360710230987123 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0958f40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:3993, start at 2025-04-04 15:44:41.043378419 +0800 CST m=+2.143042372 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:41.051 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:41.021 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:41.021 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360710230987123 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0958f40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:3993, start at 2025-04-04 15:44:41.043378419 +0800 CST m=+2.143042372 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:41.051 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:41.021 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:41.021 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360713095825554 TiDB Cluster ID. tikv_gc_leader_uuid 657ff095aa00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:4065, start at 2025-04-04 15:44:41.147624289 +0800 CST m=+2.173504194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:41.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:41.128 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:41.128 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 7-th time, retry later wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 45-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:44:45 CST 2025] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360721206040447 TiDB Cluster ID. tikv_gc_leader_uuid 657ff097eac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:9514, start at 2025-04-04 15:44:43.484723049 +0800 CST m=+2.808835158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.541 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360721206040447 TiDB Cluster ID. tikv_gc_leader_uuid 657ff097eac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:9514, start at 2025-04-04 15:44:43.484723049 +0800 CST m=+2.808835158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.541 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360716136389435 TiDB Cluster ID. tikv_gc_leader_uuid 657ff098040000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-j3m-xndqm, pid:9578, start at 2025-04-04 15:44:43.580482692 +0800 CST m=+2.822826699 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.634 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.636 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.636 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 46-th check, retry later [Fri Apr 4 15:44:45 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.53835385.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360727603241875 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0983040006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:4609, start at 2025-04-04 15:44:43.722555241 +0800 CST m=+1.981239506 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.731 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.713 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.713 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360727603241875 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0983040006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:4609, start at 2025-04-04 15:44:43.722555241 +0800 CST m=+1.981239506 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.731 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.713 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.713 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360731161362399 TiDB Cluster ID. tikv_gc_leader_uuid 657ff098540000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:4667, start at 2025-04-04 15:44:43.884415933 +0800 CST m=+2.057909534 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:43.925 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:43.913 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:43.913 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 8-th time, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114303801327620, "checkpoint_time": "2025-04-04 15:44:44.789", "error": null }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114303801327620, "checkpoint_time": "2025-04-04 15:44:44.789", "error": null }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114303801327620, "checkpoint_time": "2025-04-04 15:44:44.789", "error": null } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114303801327620, '"checkpoint_time":' '"2025-04-04' '15:44:44.789",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114303801327620, '"checkpoint_time":' '"2025-04-04' '15:44:44.789",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6586.out cli changefeed pause -c changefeed-initialize-error Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table region_merge.t1 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10883.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 47-th check, retry later check diff successfully + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114304063471621, '"checkpoint_time":' '"2025-04-04' '15:44:45.789",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114304063471621, '"checkpoint_time":' '"2025-04-04' '15:44:45.789",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6659.out cli changefeed resume -c changefeed-initialize-error + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:48 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51b0dba3-9ee2-430e-9c05-a4191c38c7f4 {"id":"51b0dba3-9ee2-430e-9c05-a4191c38c7f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752685} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc239b3dd 51b0dba3-9ee2-430e-9c05-a4191c38c7f4 /tidb/cdc/default/default/upstream/7489360710230987123 {"id":7489360710230987123,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51b0dba3-9ee2-430e-9c05-a4191c38c7f4 {"id":"51b0dba3-9ee2-430e-9c05-a4191c38c7f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752685} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc239b3dd 51b0dba3-9ee2-430e-9c05-a4191c38c7f4 /tidb/cdc/default/default/upstream/7489360710230987123 {"id":7489360710230987123,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51b0dba3-9ee2-430e-9c05-a4191c38c7f4 {"id":"51b0dba3-9ee2-430e-9c05-a4191c38c7f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752685} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc239b3dd 51b0dba3-9ee2-430e-9c05-a4191c38c7f4 /tidb/cdc/default/default/upstream/7489360710230987123 {"id":7489360710230987123,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.5449.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-21718?protocol=canal-json&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/batch_add_table/conf/changefeed.toml [Fri Apr 4 15:44:48 CST 2025] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.58945896.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: 35c15eb0-9772-4f5b-8dd9-6b4004557aae Info: {"upstream_id":7489360710230987123,"namespace":"default","id":"35c15eb0-9772-4f5b-8dd9-6b4004557aae","sink_uri":"file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-21718?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:48.794929136+08:00","start_ts":457114304832077828,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!test.t_1"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114304832077828,"checkpoint_ts":457114304832077828,"checkpoint_time":"2025-04-04 15:44:48.721"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114304495222789 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114304495222789 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:48 CST 2025] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1092510927.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 48-th check, retry later table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later + workdir=/tmp/tidb_cdc_test/canal_json_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_partition_table ++ date + echo '[Fri Apr 4 15:44:49 CST 2025] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>>' [Fri Apr 4 15:44:49 CST 2025] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 2-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + set +x + workdir=/tmp/tidb_cdc_test/batch_add_table + sink_uri='file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-21718?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/batch_add_table ++ date + echo '[Fri Apr 4 15:44:50 CST 2025] <<<<<< START storage consumer in batch_add_table case >>>>>>' [Fri Apr 4 15:44:50 CST 2025] <<<<<< START storage consumer in batch_add_table case >>>>>> + cd /tmp/tidb_cdc_test/batch_add_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/batch_add_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/batch_add_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-21718?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + info='{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null }' + echo '{ "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null }' { "upstream_id": 7489360427031806945, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114304063471621, "checkpoint_time": "2025-04-04 15:44:45.789", "error": null } ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114304063471621, '"checkpoint_time":' '"2025-04-04' '15:44:45.789",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360427031806945, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114304063471621, '"checkpoint_time":' '"2025-04-04' '15:44:45.789",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6729.out cli changefeed remove -c changefeed-initialize-error table test.t_1 not exists for 1-th check, retry later Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 457114304063471621 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-11084?protocol=canal-json&enable-tidb-extension=true PASS check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 49-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:51 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15e2d3a0-4120-4517-a526-5f813a275070 {"id":"15e2d3a0-4120-4517-a526-5f813a275070","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc248dfd6 15e2d3a0-4120-4517-a526-5f813a275070 /tidb/cdc/default/default/upstream/7489360727603241875 {"id":7489360727603241875,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15e2d3a0-4120-4517-a526-5f813a275070 {"id":"15e2d3a0-4120-4517-a526-5f813a275070","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc248dfd6 15e2d3a0-4120-4517-a526-5f813a275070 /tidb/cdc/default/default/upstream/7489360727603241875 {"id":7489360727603241875,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15e2d3a0-4120-4517-a526-5f813a275070 {"id":"15e2d3a0-4120-4517-a526-5f813a275070","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc248dfd6 15e2d3a0-4120-4517-a526-5f813a275070 /tidb/cdc/default/default/upstream/7489360727603241875 {"id":7489360727603241875,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_etcd_worker_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-4753?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_etcd_worker_delay ++ date + echo '[Fri Apr 4 15:44:51 CST 2025] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>>' [Fri Apr 4 15:44:51 CST 2025] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-4753?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.cli.9625.out cli changefeed create --start-ts=457114297101975559 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed3?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed3.toml -c changefeed3 [2025/04/04 15:44:51.847 +08:00] [WARN] [sink.go:900] ["TiCDC will not split the update pk/uk events if output-raw-change-event is true(scheme: file).It is not recommended to set output-old-value to false(protocol: csv) in this case.Otherwise, there may be data consistency issues in update pk/uk scenarios due to lack of old value information."] Create changefeed successfully! ID: changefeed3 Info: {"upstream_id":7489360589440889998,"namespace":"default","id":"changefeed3","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed3?flush-interval=5s","create_time":"2025-04-04T15:44:51.910454165+08:00","start_ts":457114297101975559,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":true},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297101975559,"checkpoint_ts":457114297101975559,"checkpoint_time":"2025-04-04 15:44:19.233"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:52 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23068f66-fa86-4545-83d4-154d1c70483c {"id":"23068f66-fa86-4545-83d4-154d1c70483c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23ecbe2 23068f66-fa86-4545-83d4-154d1c70483c /tidb/cdc/default/default/upstream/7489360721206040447 {"id":7489360721206040447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23068f66-fa86-4545-83d4-154d1c70483c {"id":"23068f66-fa86-4545-83d4-154d1c70483c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23ecbe2 23068f66-fa86-4545-83d4-154d1c70483c /tidb/cdc/default/default/upstream/7489360721206040447 {"id":7489360721206040447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23068f66-fa86-4545-83d4-154d1c70483c {"id":"23068f66-fa86-4545-83d4-154d1c70483c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23ecbe2 23068f66-fa86-4545-83d4-154d1c70483c /tidb/cdc/default/default/upstream/7489360721206040447 {"id":7489360721206040447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10988.out cli changefeed create --start-ts=457114304495222789 '--sink-uri=file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-672?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: a8e2f2ae-4340-48ed-b08d-0ad83ff5b20b Info: {"upstream_id":7489360721206040447,"namespace":"default","id":"a8e2f2ae-4340-48ed-b08d-0ad83ff5b20b","sink_uri":"file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-672?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:52.519145518+08:00","start_ts":457114304495222789,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114304495222789,"checkpoint_ts":457114304495222789,"checkpoint_time":"2025-04-04 15:44:47.436"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x table test.t_1 exists table test.t_2 exists table test.t_3 exists table test.t_4 exists table test.t_5 exists table test.t_6 exists table test.t_7 exists table test.t_8 exists table test.t_9 exists table test.t_10 exists table test.t_11 exists table test.t_12 exists table test.t_13 exists table test.t_14 exists table test.t_15 exists table test.t_16 exists table test.t_17 exists table test.t_18 exists table test.t_19 exists table test.t_20 exists table test.t_21 exists table test.t_22 exists table test.t_23 exists table test.t_24 exists table test.t_25 exists table test.t_26 exists table test.t_27 exists table test.t_28 exists table test.t_29 exists table test.t_30 exists table test.t_31 exists table test.t_32 exists table test.t_33 exists table test.t_34 exists table test.t_35 exists table test.t_36 exists table test.t_37 exists table test.t_38 exists check diff failed 4-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... check diff failed 3-th time, retry later table test.t_39 exists table test.t_40 exists table test.t_41 exists table test.t_42 exists table test.t_43 exists table test.t_44 exists table test.t_45 exists table test.t_46 exists table test.t_47 exists table test.t_48 exists table test.t_49 exists table test.t_50 exists table test.t_51 exists table test.t_52 exists table test.t_53 exists table test.t_54 exists table test.t_55 exists table test.t_56 exists table test.t_57 exists table test.t_58 exists table test.t_59 exists table test.t_60 exists table test.t_61 exists table test.t_62 exists table test.t_63 exists table test.t_64 exists table test.t_65 exists table test.t_66 exists wait process cdc.test exit for 2-th time... table test.t_67 exists table test.t_68 exists table test.t_69 exists table test.t_70 exists table test.t_71 exists table test.t_72 exists table test.t_73 exists table test.t_74 exists table test.t_75 exists table test.t_76 exists table test.t_77 exists table test.t_78 exists table test.t_79 exists table test.t_80 exists table test.t_81 exists table test.t_82 exists table test.t_83 exists table test.t_84 exists table test.t_85 exists table test.t_86 exists table test.t_87 exists table test.t_88 exists table test.t_89 exists table test.t_90 exists table test.t_91 exists table test.t_92 exists table test.t_93 exists table test.t_94 exists table test.t_95 exists table test.t_96 exists table ddl_manager.finish_mark not exists for 50-th check, retry later table test.t_97 exists table test.t_98 exists table test.t_99 exists table test.t_100 exists + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed3?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed3.toml + log_suffix=changefeed3 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered ++ date + echo '[Fri Apr 4 15:44:53 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>>' [Fri Apr 4 15:44:53 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed3.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc_storage_consumerchangefeed3.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed3?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed3.toml table ddl_attributes.finish_mark not exists for 2-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/partition_table + sink_uri='file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-672?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/partition_table ++ date + echo '[Fri Apr 4 15:44:53 CST 2025] <<<<<< START storage consumer in partition_table case >>>>>>' [Fri Apr 4 15:44:53 CST 2025] <<<<<< START storage consumer in partition_table case >>>>>> + cd /tmp/tidb_cdc_test/partition_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-672?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:53 CST 2025] <<<<<< run test case changefeed_error success! >>>>>> table test.t_100 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore check diff failed 4-th time, retry later Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully table ddl_manager.finish_mark not exists for 51-th check, retry later wait process cdc.test exit for 1-th time... table ddl_attributes.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:56 CST 2025] <<<<<< run test case region_merge success! >>>>>> table test.t_100 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 52-th check, retry later invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 4 15:44:57 CST 2025] <<<<<< run all test cases cdc_server_tips success! >>>>>> check diff failed 5-th time, retry later table ddl_attributes.finish_mark not exists for 4-th check, retry later table partition_table.t not exists for 1-th check, retry later table test.t_100 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 53-th check, retry later check diff failed 6-th time, retry later table ddl_attributes.finish_mark not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360780878824220 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a5554000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:10010, start at 2025-04-04 15:44:57.183204417 +0800 CST m=+2.228355931 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:57.192 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:57.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:57.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360780878824220 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a5554000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:10010, start at 2025-04-04 15:44:57.183204417 +0800 CST m=+2.228355931 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:57.192 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:57.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:57.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360772022078708 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0a5c58000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:10070, start at 2025-04-04 15:44:57.670670382 +0800 CST m=+2.651019999 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:57.678 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:57.671 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:57.671 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.t not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table test.t_100 not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 54-th check, retry later table ddl_attributes.finish_mark not exists for 6-th check, retry later + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11456.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later check diff failed 7-th time, retry later table partition_table.t exists table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later table test.t_100 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x + tso='457114308240474113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114308240474113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114308240474113 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:45:03 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1149511497.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 55-th check, retry later check diff failed 2-th time, retry later check diff failed 8-th time, retry later table ddl_attributes.finish_mark not exists for 7-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later table test.t_100 not exists for 6-th check, retry later table ddl_manager.finish_mark not exists for 56-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 9-th time, retry later table ddl_attributes.finish_mark not exists for 8-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later check diff successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_clustered.cli.9822.out cli changefeed create --start-ts=457114297101975559 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed4?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed4.toml -c changefeed4 Create changefeed successfully! ID: changefeed4 Info: {"upstream_id":7489360589440889998,"namespace":"default","id":"changefeed4","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed4?flush-interval=5s","create_time":"2025-04-04T15:45:06.257332776+08:00","start_ts":457114297101975559,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":true,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":false},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297101975559,"checkpoint_ts":457114297101975559,"checkpoint_time":"2025-04-04 15:44:19.233"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:06 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6343e3a3-a352-417d-b03d-40d7516a78cd {"id":"6343e3a3-a352-417d-b03d-40d7516a78cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752703} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc276a6e1 6343e3a3-a352-417d-b03d-40d7516a78cd /tidb/cdc/default/default/upstream/7489360780878824220 {"id":7489360780878824220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6343e3a3-a352-417d-b03d-40d7516a78cd {"id":"6343e3a3-a352-417d-b03d-40d7516a78cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752703} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc276a6e1 6343e3a3-a352-417d-b03d-40d7516a78cd /tidb/cdc/default/default/upstream/7489360780878824220 {"id":7489360780878824220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6343e3a3-a352-417d-b03d-40d7516a78cd {"id":"6343e3a3-a352-417d-b03d-40d7516a78cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752703} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc276a6e1 6343e3a3-a352-417d-b03d-40d7516a78cd /tidb/cdc/default/default/upstream/7489360780878824220 {"id":7489360780878824220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114308240474113 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11548.out cli changefeed create --start-ts=457114308240474113 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360780878824220,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:45:06.642649472+08:00","start_ts":457114308240474113,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114308240474113,"checkpoint_ts":457114308240474113,"checkpoint_time":"2025-04-04 15:45:01.723"} PASS table test.t_100 not exists for 7-th check, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 57-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 9-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed4?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed4.toml + log_suffix=changefeed4 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_clustered ++ date + echo '[Fri Apr 4 15:45:07 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>>' [Fri Apr 4 15:45:07 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_clustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed4.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_clustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_clustered/cdc_storage_consumerchangefeed4.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_clustered/storage_test/changefeed4?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/conf/changefeed4.toml + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 5022 0 --:--:-- --:--:-- --:--:-- 5022 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:45:01.723","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:45:08.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:45:01.723","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:45:08.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:45:01.723","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:45:08.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:45:01.723' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:45:01.723","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:45:08.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:45:01.723","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:45:08.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:45:08' + echo 'sink_checkpoint_ts is 2025-04-04' 15:45:01.723 sink_checkpoint_ts is 2025-04-04 15:45:01.723 ++ date -d '2025-04-04 15:45:01.723' +%s + checkpoint_timestamp=1743752701 ++ date -d '2025-04-04 15:45:08' +%s + current_timestamp=1743752708 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' check diff successfully table test.t_100 not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:45:08 CST 2025] <<<<<< run test case common_1 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 58-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 10-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table partition_table.finish_mark not exists for 5-th check, retry later table test.t_100 not exists for 9-th check, retry later table test.t1 exists + sleep 5 check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 59-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360838651452999 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b13b40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:10809, start at 2025-04-04 15:45:09.358881417 +0800 CST m=+2.128475315 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:09.367 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:09.357 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:09.357 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360838651452999 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b13b40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:10809, start at 2025-04-04 15:45:09.358881417 +0800 CST m=+2.128475315 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:09.367 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:09.357 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:09.357 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360840163919037 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0b15280005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-wzc-5fr9w, pid:10875, start at 2025-04-04 15:45:09.45768208 +0800 CST m=+2.129113600 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:09.465 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:09.450 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:09.450 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_attributes.finish_mark exists check diff successfully \033[0;36m<<< Run all test success >>>\033[0m table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later table partition_table.t2 not exists for 2-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } check_changefeed_state http://127.0.0.1:2379 a967ebe3-b3b9-4004-8985-a52780618707 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=a967ebe3-b3b9-4004-8985-a52780618707 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c a967ebe3-b3b9-4004-8985-a52780618707 -s + info='{ "upstream_id": 7489360354056934566, "namespace": "default", "id": "a967ebe3-b3b9-4004-8985-a52780618707", "state": "finished", "checkpoint_tso": 457114306376892420, "checkpoint_time": "2025-04-04 15:44:54.614", "error": null }' + echo '{ "upstream_id": 7489360354056934566, "namespace": "default", "id": "a967ebe3-b3b9-4004-8985-a52780618707", "state": "finished", "checkpoint_tso": 457114306376892420, "checkpoint_time": "2025-04-04 15:44:54.614", "error": null }' { "upstream_id": 7489360354056934566, "namespace": "default", "id": "a967ebe3-b3b9-4004-8985-a52780618707", "state": "finished", "checkpoint_tso": 457114306376892420, "checkpoint_time": "2025-04-04 15:44:54.614", "error": null } ++ echo '{' '"upstream_id":' 7489360354056934566, '"namespace":' '"default",' '"id":' '"a967ebe3-b3b9-4004-8985-a52780618707",' '"state":' '"finished",' '"checkpoint_tso":' 457114306376892420, '"checkpoint_time":' '"2025-04-04' '15:44:54.614",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7489360354056934566, '"namespace":' '"default",' '"id":' '"a967ebe3-b3b9-4004-8985-a52780618707",' '"state":' '"finished",' '"checkpoint_tso":' 457114306376892420, '"checkpoint_time":' '"2025-04-04' '15:44:54.614",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:45:07 CST 2025] <<<<<< run test case changefeed_finish success! >>>>>> [Pipeline] // dir [Pipeline] } table partition_table.finish_mark not exists for 6-th check, retry later [Pipeline] // withCredentials [Pipeline] } table test.t_100 not exists for 10-th check, retry later wait process cdc.test exit for 1-th time... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit table partition_table.t2 not exists for 3-th check, retry later [Pipeline] // stage [Pipeline] } [Fri Apr 4 15:45:13 CST 2025] <<<<<< run test case ddl_attributes success! >>>>>> check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 60-th check, retry later [Fri Apr 4 15:45:13 CST 2025] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1213212135.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data0 --cluster-id default --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 7-th check, retry later table test.t_100 not exists for 11-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 61-th check, retry later table partition_table.t2 not exists for 4-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 5522 0 --:--:-- --:--:-- --:--:-- 5522 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:45:14.024","puller_resolved_ts":"2025-04-04 15:45:08.173","last_synced_ts":"2025-04-04 15:45:08.673","now_ts":"2025-04-04 15:45:15.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:14.024","puller_resolved_ts":"2025-04-04' '15:45:08.173","last_synced_ts":"2025-04-04' '15:45:08.673","now_ts":"2025-04-04' '15:45:15.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:14.024","puller_resolved_ts":"2025-04-04' '15:45:08.173","last_synced_ts":"2025-04-04' '15:45:08.673","now_ts":"2025-04-04' '15:45:15.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table partition_table.finish_mark not exists for 8-th check, retry later check diff failed 1-th time, retry later Starting Upstream TiDB... table test.t_100 not exists for 12-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:16 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:45:16 CST 2025] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1219712199.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 62-th check, retry later table partition_table.t2 not exists for 5-th check, retry later table partition_table.finish_mark not exists for 9-th check, retry later check diff failed 5-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore table test.t_100 not exists for 13-th check, retry later Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 63-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 6-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:19 GMT < Content-Length: 1277 < Content-Type: text/plain; charset=utf-8 < { [1277 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:45:19 CST 2025] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1225612258.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 6-th time, retry later check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Fri Apr 4 15:45:20 CST 2025] <<<<<< run test case csv_storage_update_pk_clustered success! >>>>>> table test.t_100 not exists for 14-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 64-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360875452754244 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bac68000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:7717, start at 2025-04-04 15:45:19.149234229 +0800 CST m=+2.294643934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:19.157 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:19.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:19.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360875452754244 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bac68000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:7717, start at 2025-04-04 15:45:19.149234229 +0800 CST m=+2.294643934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:19.157 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:19.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:19.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360881856742197 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0badf8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:7787, start at 2025-04-04 15:45:19.276044305 +0800 CST m=+2.324861861 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:19.283 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:19.276 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:19.276 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.t2 not exists for 7-th check, retry later check diff failed 7-th time, retry later table partition_table.finish_mark not exists for 11-th check, retry later table test.t_100 not exists for 15-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360887203026434 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bc8a40005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:6414, start at 2025-04-04 15:45:20.94485791 +0800 CST m=+2.425305638 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:20.954 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:20.992 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:20.992 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360887203026434 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bc8a40005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:6414, start at 2025-04-04 15:45:20.94485791 +0800 CST m=+2.425305638 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:20.954 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:20.992 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:20.992 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360877410520002 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bcb100009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:6493, start at 2025-04-04 15:45:21.108228368 +0800 CST m=+2.483743207 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:21.117 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:21.092 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:21.092 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:22 GMT < Content-Length: 1759 < Content-Type: text/plain; charset=utf-8 < { [1759 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/4f0c3a41-0bcc-4217-b48d-76a266e8d48f {"id":"4f0c3a41-0bcc-4217-b48d-76a266e8d48f","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89536 4f0c3a41-0bcc-4217-b48d-76a266e8d48f /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/4f0c3a41-0bcc-4217-b48d-76a266e8d48f {"id":"4f0c3a41-0bcc-4217-b48d-76a266e8d48f","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89536 4f0c3a41-0bcc-4217-b48d-76a266e8d48f /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1378f12f-078c-44b0-be63-b5d573c65515 {"id":"1378f12f-078c-44b0-be63-b5d573c65515","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752717} /tidb/cdc/default/__cdc_meta__/capture/4f0c3a41-0bcc-4217-b48d-76a266e8d48f {"id":"4f0c3a41-0bcc-4217-b48d-76a266e8d48f","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752720} /tidb/cdc/default/__cdc_meta__/capture/8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 {"id":"8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752714} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a894d5 8a18d6bc-d842-41a0-b7b0-bb47f6ac55f7 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89515 1378f12f-078c-44b0-be63-b5d573c65515 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2a89536 4f0c3a41-0bcc-4217-b48d-76a266e8d48f /tidb/cdc/default/default/upstream/7489360838651452999 {"id":7489360838651452999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.cli.12307.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.9112.out cli tso query --pd=http://127.0.0.1:2379 table partition_table.t2 not exists for 8-th check, retry later sink.enable-partition-separator is deprecated and will be removed in future versions. The default value for sink.enable-partition-separator is `true`. Changing it to `false` may lead to data inconsistency issues. Please update your configuration as needed. Create changefeed successfully! ID: 034c1791-2990-4ea4-a5a4-95f92235f43c Info: {"upstream_id":7489360838651452999,"namespace":"default","id":"034c1791-2990-4ea4-a5a4-95f92235f43c","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:45:23.285809342+08:00","start_ts":457114313859530767,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114313859530767,"checkpoint_ts":457114313859530767,"checkpoint_time":"2025-04-04 15:45:23.158"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 65-th check, retry later table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later table partition_table.finish_mark not exists for 12-th check, retry later check diff failed 8-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x table test.t_100 not exists for 16-th check, retry later [Fri Apr 4 15:45:24 CST 2025] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.77947796.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114313969631233 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114313969631233 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:45:25 CST 2025] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.91389140.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 66-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) table partition_table.t2 not exists for 9-th check, retry later [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } table partition_table.finish_mark not exists for 13-th check, retry later [Pipeline] // stage [Pipeline] } check diff failed 9-th time, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table processor_delay.t6 not exists for 2-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table test.t_100 not exists for 17-th check, retry later [Pipeline] // stage [Pipeline] } table ddl_manager.finish_mark not exists for 67-th check, retry later table partition_table.t2 not exists for 10-th check, retry later table partition_table.finish_mark not exists for 14-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:28 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/be877b94-7f0f-4f2a-ba37-bac5f23e00a0 {"id":"be877b94-7f0f-4f2a-ba37-bac5f23e00a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d2afd5 be877b94-7f0f-4f2a-ba37-bac5f23e00a0 /tidb/cdc/default/default/upstream/7489360887203026434 {"id":7489360887203026434,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/be877b94-7f0f-4f2a-ba37-bac5f23e00a0 {"id":"be877b94-7f0f-4f2a-ba37-bac5f23e00a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d2afd5 be877b94-7f0f-4f2a-ba37-bac5f23e00a0 /tidb/cdc/default/default/upstream/7489360887203026434 {"id":7489360887203026434,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/be877b94-7f0f-4f2a-ba37-bac5f23e00a0 {"id":"be877b94-7f0f-4f2a-ba37-bac5f23e00a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2d2afd5 be877b94-7f0f-4f2a-ba37-bac5f23e00a0 /tidb/cdc/default/default/upstream/7489360887203026434 {"id":7489360887203026434,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7848.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-29386?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml table processor_delay.t6 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:28 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50273a9c-ffca-440f-84d7-bca1ad0c99d9 {"id":"50273a9c-ffca-440f-84d7-bca1ad0c99d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2cc36e4 50273a9c-ffca-440f-84d7-bca1ad0c99d9 /tidb/cdc/default/default/upstream/7489360875452754244 {"id":7489360875452754244,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50273a9c-ffca-440f-84d7-bca1ad0c99d9 {"id":"50273a9c-ffca-440f-84d7-bca1ad0c99d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2cc36e4 50273a9c-ffca-440f-84d7-bca1ad0c99d9 /tidb/cdc/default/default/upstream/7489360875452754244 {"id":7489360875452754244,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50273a9c-ffca-440f-84d7-bca1ad0c99d9 {"id":"50273a9c-ffca-440f-84d7-bca1ad0c99d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752725} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2cc36e4 50273a9c-ffca-440f-84d7-bca1ad0c99d9 /tidb/cdc/default/default/upstream/7489360875452754244 {"id":7489360875452754244,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.9196.out cli changefeed create --start-ts=457114313969631233 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-12185?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 1008e2ba-4153-4af4-9200-7a678f5622a4 Info: {"upstream_id":7489360887203026434,"namespace":"default","id":"1008e2ba-4153-4af4-9200-7a678f5622a4","sink_uri":"file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-29386?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:45:28.496324008+08:00","start_ts":457114315230281732,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_name*"],"ignore_event":["rename table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114315230281732,"checkpoint_ts":457114315230281732,"checkpoint_time":"2025-04-04 15:45:28.387"} PASS Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Create changefeed successfully! ID: c2e8ea8f-1e5f-4d50-a523-82eb9c673b46 Info: {"upstream_id":7489360875452754244,"namespace":"default","id":"c2e8ea8f-1e5f-4d50-a523-82eb9c673b46","sink_uri":"file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-12185?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:45:28.523270793+08:00","start_ts":457114313969631233,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114313969631233,"checkpoint_ts":457114313969631233,"checkpoint_time":"2025-04-04 15:45:23.578"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table test.t_100 not exists for 18-th check, retry later coverage: 2.6% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 68-th check, retry later check diff failed 1-th time, retry later table partition_table.t2 not exists for 11-th check, retry later Starting Upstream TiDB... table partition_table.finish_mark not exists for 15-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/ddl_sequence + sink_uri='file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-12185?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_sequence ++ date + echo '[Fri Apr 4 15:45:29 CST 2025] <<<<<< START storage consumer in ddl_sequence case >>>>>>' [Fri Apr 4 15:45:29 CST 2025] <<<<<< START storage consumer in ddl_sequence case >>>>>> + cd /tmp/tidb_cdc_test/ddl_sequence + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_sequence + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-12185?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + workdir=/tmp/tidb_cdc_test/event_filter + sink_uri='file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-29386?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/event_filter ++ date + echo '[Fri Apr 4 15:45:29 CST 2025] <<<<<< START storage consumer in event_filter case >>>>>>' [Fri Apr 4 15:45:29 CST 2025] <<<<<< START storage consumer in event_filter case >>>>>> + cd /tmp/tidb_cdc_test/event_filter + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/event_filter + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/event_filter/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-29386?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t6 not exists for 4-th check, retry later table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later table test.t_100 not exists for 19-th check, retry later table ddl_sequence.finish_mark not exists for 1-th check, retry later check diff failed 2-th time, retry later table partition_table.t2 not exists for 12-th check, retry later table ddl_manager.finish_mark not exists for 69-th check, retry later table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later table partition_table.finish_mark not exists for 16-th check, retry later table test.t_100 not exists for 20-th check, retry later table ddl_sequence.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t1 not exists for 2-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 4-th time, retry later table ddl_manager.finish_mark not exists for 70-th check, retry later check diff failed 3-th time, retry later table partition_table.t2 not exists for 13-th check, retry later table processor_delay.t8 not exists for 2-th check, retry later table partition_table.finish_mark exists check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360933448362863 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c78a00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:4114, start at 2025-04-04 15:45:32.234862379 +0800 CST m=+2.304910844 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:32.243 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:32.246 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:32.246 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360933448362863 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c78a00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:4114, start at 2025-04-04 15:45:32.234862379 +0800 CST m=+2.304910844 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:32.243 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:32.246 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:32.246 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360925997702313 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c7bc0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-hfd-gc4mp, pid:4193, start at 2025-04-04 15:45:32.444652417 +0800 CST m=+2.416311864 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:32.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:32.444 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:32.444 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... table event_filter.t1 not exists for 3-th check, retry later table test.t_100 not exists for 21-th check, retry later table ddl_sequence.finish_mark not exists for 3-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 5-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 71-th check, retry later wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:35 CST 2025] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> check diff failed 4-th time, retry later table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later table processor_delay.t8 not exists for 3-th check, retry later table event_filter.t1 not exists for 4-th check, retry later table test.t_100 exists table ddl_sequence.finish_mark not exists for 4-th check, retry later [Fri Apr 4 15:45:36 CST 2025] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.55275529.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 72-th check, retry later check data successfully table partition_table.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 5-th time, retry later wait process cdc.test exit for 2-th time... table processor_delay.t8 not exists for 4-th check, retry later table event_filter.t1 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:38 CST 2025] <<<<<< run test case ddl_puller_lag success! >>>>>> table ddl_manager.finish_mark not exists for 73-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:39 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56ea45a7-70b6-4502-9823-2f7bcc173e05 {"id":"56ea45a7-70b6-4502-9823-2f7bcc173e05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752736} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2fd4ed9 56ea45a7-70b6-4502-9823-2f7bcc173e05 /tidb/cdc/default/default/upstream/7489360933448362863 {"id":7489360933448362863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56ea45a7-70b6-4502-9823-2f7bcc173e05 {"id":"56ea45a7-70b6-4502-9823-2f7bcc173e05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752736} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2fd4ed9 56ea45a7-70b6-4502-9823-2f7bcc173e05 /tidb/cdc/default/default/upstream/7489360933448362863 {"id":7489360933448362863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56ea45a7-70b6-4502-9823-2f7bcc173e05 {"id":"56ea45a7-70b6-4502-9823-2f7bcc173e05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752736} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2fd4ed9 56ea45a7-70b6-4502-9823-2f7bcc173e05 /tidb/cdc/default/default/upstream/7489360933448362863 {"id":7489360933448362863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' table partition_table.finish_mark not exists for 3-th check, retry later + break + set +x Create changefeed successfully! ID: 3dbb4783-b70b-4dee-b127-a72053e8f084 Info: {"upstream_id":7489360933448362863,"namespace":"default","id":"3dbb4783-b70b-4dee-b127-a72053e8f084","sink_uri":"file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-19918?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:45:39.64805188+08:00","start_ts":457114317329793025,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114317329793025,"checkpoint_ts":457114317329793025,"checkpoint_time":"2025-04-04 15:45:36.396"} + workdir=/tmp/tidb_cdc_test/force_replicate_table + sink_uri='file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-19918?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/force_replicate_table ++ date + echo '[Fri Apr 4 15:45:39 CST 2025] <<<<<< START storage consumer in force_replicate_table case >>>>>>' [Fri Apr 4 15:45:39 CST 2025] <<<<<< START storage consumer in force_replicate_table case >>>>>> + cd /tmp/tidb_cdc_test/force_replicate_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/force_replicate_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-19918?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml check diff failed 6-th time, retry later table processor_delay.t8 not exists for 5-th check, retry later table force_replicate_table.t0 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_sequence.finish_mark not exists for 6-th check, retry later table event_filter.t1 exists table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 74-th check, retry later check diff failed 7-th time, retry later table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 7-th check, retry later table force_replicate_table.t0 not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 75-th check, retry later table partition_table.finish_mark not exists for 5-th check, retry later check diff failed 8-th time, retry later table processor_delay.t10 not exists for 2-th check, retry later table force_replicate_table.t0 not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t_alter not exists for 2-th check, retry later table ddl_sequence.finish_mark not exists for 8-th check, retry later table ddl_manager.finish_mark not exists for 76-th check, retry later table partition_table.finish_mark not exists for 6-th check, retry later check diff failed 9-th time, retry later table processor_delay.t10 not exists for 3-th check, retry later table force_replicate_table.t0 not exists for 4-th check, retry later table event_filter.t_alter not exists for 3-th check, retry later table batch_add_table.finish_mark not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 77-th check, retry later table batch_add_table.finish_mark not exists for 2-th check, retry later table partition_table.finish_mark not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: storage... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table processor_delay.t10 not exists for 4-th check, retry later table force_replicate_table.t0 not exists for 5-th check, retry later table event_filter.t_alter not exists for 4-th check, retry later table ddl_sequence.finish_mark not exists for 10-th check, retry later check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360998406299861 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d637c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:11005, start at 2025-04-04 15:45:47.280414701 +0800 CST m=+2.666886436 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:47.288 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:47.331 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:47.331 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360998406299861 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d637c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:11005, start at 2025-04-04 15:45:47.280414701 +0800 CST m=+2.666886436 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:47.288 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:47.331 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:47.331 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361003276616209 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0d6768000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-jl8-z73vx, pid:11070, start at 2025-04-04 15:45:47.503070394 +0800 CST m=+2.818387508 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:47.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:47.532 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:47.532 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 78-th check, retry later table partition_table.finish_mark not exists for 8-th check, retry later * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DE3075E49B6 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:45:49 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact check diff failed 1-th time, retry later table batch_add_table.finish_mark not exists for 3-th check, retry later table processor_delay.t10 not exists for 5-th check, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists Bucket 's3://logbucket/' created [Fri Apr 4 15:45:50 CST 2025] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT table force_replicate_table.t3 exists table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 11-th check, retry later table event_filter.t_alter exists table event_filter.t_name exists table event_filter.t_name1 exists table event_filter.t_name2 exists table event_filter.t_name3 exists + workdir=/tmp/tidb_cdc_test/csv_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_partition_table ++ date + echo '[Fri Apr 4 15:45:47 CST 2025] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>>' [Fri Apr 4 15:45:47 CST 2025] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml table partition_table.finish_mark not exists for 9-th check, retry later table event_filter.finish_mark exists [Fri Apr 4 15:45:51 CST 2025] <<<<<< START cdc server in csv_storage_update_pk_nonclustered case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.1230912311.out server --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 79-th check, retry later table batch_add_table.finish_mark not exists for 4-th check, retry later table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later table force_replicate_table.t5 not exists for 2-th check, retry later check diff failed 2-th time, retry later table ddl_sequence.finish_mark not exists for 12-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 80-th check, retry later table batch_add_table.finish_mark not exists for 5-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:54 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39336b08-2f06-4853-8ac4-42660c93880d {"id":"39336b08-2f06-4853-8ac4-42660c93880d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752751} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33a92d5 39336b08-2f06-4853-8ac4-42660c93880d /tidb/cdc/default/default/upstream/7489360998406299861 {"id":7489360998406299861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39336b08-2f06-4853-8ac4-42660c93880d {"id":"39336b08-2f06-4853-8ac4-42660c93880d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752751} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33a92d5 39336b08-2f06-4853-8ac4-42660c93880d /tidb/cdc/default/default/upstream/7489360998406299861 {"id":7489360998406299861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39336b08-2f06-4853-8ac4-42660c93880d {"id":"39336b08-2f06-4853-8ac4-42660c93880d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752751} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc33a92d5 39336b08-2f06-4853-8ac4-42660c93880d /tidb/cdc/default/default/upstream/7489360998406299861 {"id":7489360998406299861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.cli.12370.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 2-th time, retry later check diff failed 3-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table processor_delay.t12 not exists for 2-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } table force_replicate_table.t5 not exists for 3-th check, retry later [Pipeline] // timeout [Pipeline] } table ddl_sequence.finish_mark not exists for 13-th check, retry later [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table ddl_manager.finish_mark not exists for 81-th check, retry later table batch_add_table.finish_mark not exists for 6-th check, retry later table partition_table.finish_mark not exists for 11-th check, retry later + set +x + tso='457114322070667269 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114322070667269 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.cli.12417.out cli changefeed create --start-ts=457114322070667269 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed1?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed1.toml -c changefeed1 table partition_table.t not exists for 1-th check, retry later table processor_delay.t12 not exists for 3-th check, retry later Create changefeed successfully! ID: changefeed1 table force_replicate_table.t5 not exists for 4-th check, retry later Info: {"upstream_id":7489360998406299861,"namespace":"default","id":"changefeed1","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed1?flush-interval=5s","create_time":"2025-04-04T15:45:56.633133259+08:00","start_ts":457114322070667269,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":false},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114322070667269,"checkpoint_ts":457114322070667269,"checkpoint_time":"2025-04-04 15:45:54.481"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 4-th time, retry later table ddl_sequence.finish_mark not exists for 14-th check, retry later table ddl_manager.finish_mark not exists for 82-th check, retry later table batch_add_table.finish_mark not exists for 7-th check, retry later table partition_table.finish_mark not exists for 12-th check, retry later table partition_table.t not exists for 2-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed1?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed1.toml + log_suffix=changefeed1 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered ++ date + echo '[Fri Apr 4 15:45:58 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>>' [Fri Apr 4 15:45:58 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed1.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc_storage_consumerchangefeed1.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed1?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed1.toml table processor_delay.t12 not exists for 4-th check, retry later check diff failed 4-th time, retry later table force_replicate_table.t5 not exists for 5-th check, retry later check diff failed 5-th time, retry later table ddl_sequence.finish_mark not exists for 15-th check, retry later table ddl_manager.finish_mark not exists for 83-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table partition_table.finish_mark not exists for 13-th check, retry later table partition_table.t exists table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 8-th check, retry later table processor_delay.t12 not exists for 5-th check, retry later check diff failed 5-th time, retry later table force_replicate_table.t5 not exists for 6-th check, retry later check diff failed 6-th time, retry later table ddl_sequence.finish_mark not exists for 16-th check, retry later table partition_table.finish_mark not exists for 14-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark exists check diff successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 1-th time... table processor_delay.t12 exists table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later check diff failed 6-th time, retry later table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 17-th check, retry later wait process cdc.test exit for 2-th time... check diff failed 7-th time, retry later wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:46:03 CST 2025] <<<<<< run test case ddl_manager success! >>>>>> table partition_table.finish_mark not exists for 15-th check, retry later table batch_add_table.finish_mark not exists for 10-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later table processor_delay.t14 not exists for 2-th check, retry later check diff failed 7-th time, retry later table force_replicate_table.t6 not exists for 2-th check, retry later check diff failed 8-th time, retry later table ddl_sequence.finish_mark not exists for 18-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361070460815416 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e66180004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:6859, start at 2025-04-04 15:46:03.788919713 +0800 CST m=+2.216069819 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:03.838 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:03.833 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:03.833 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361070460815416 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e66180004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:6859, start at 2025-04-04 15:46:03.788919713 +0800 CST m=+2.216069819 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:03.838 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:03.833 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:03.833 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361068203710741 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0e654c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:6916, start at 2025-04-04 15:46:03.780487644 +0800 CST m=+2.126294464 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:03.788 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:03.832 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:03.832 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table partition_table.finish_mark exists check diff successfully table batch_add_table.finish_mark not exists for 11-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... table processor_delay.t14 not exists for 3-th check, retry later check diff failed 8-th time, retry later table force_replicate_table.t6 not exists for 3-th check, retry later table ddl_sequence.finish_mark not exists for 19-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... check diff failed 9-th time, retry later table batch_add_table.finish_mark not exists for 12-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:46:07 CST 2025] <<<<<< run test case partition_table success! >>>>>> table partition_table.finish_mark not exists for 5-th check, retry later table processor_delay.t14 not exists for 4-th check, retry later [Fri Apr 4 15:46:07 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.81598161.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table force_replicate_table.t6 not exists for 4-th check, retry later check diff failed 2-th time, retry later table ddl_sequence.finish_mark not exists for 20-th check, retry later check diff failed 9-th time, retry later check diff successfully table partition_table.finish_mark not exists for 6-th check, retry later table batch_add_table.finish_mark not exists for 13-th check, retry later table processor_delay.t14 not exists for 5-th check, retry later check diff failed 1-th time, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 check diff successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.cli.12631.out cli changefeed create --start-ts=457114322070667269 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed2?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed2.toml -c changefeed2 table ddl_sequence.finish_mark not exists for 21-th check, retry later Create changefeed successfully! ID: changefeed2 Info: {"upstream_id":7489360998406299861,"namespace":"default","id":"changefeed2","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed2?flush-interval=5s","create_time":"2025-04-04T15:46:10.832536445+08:00","start_ts":457114322070667269,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":true,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":true},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114322070667269,"checkpoint_ts":457114322070667269,"checkpoint_time":"2025-04-04 15:45:54.481"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > check diff failed 10-th time, retry later < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:11 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0301ac6f-7f8e-4a41-a36e-65eb0d382584 {"id":"0301ac6f-7f8e-4a41-a36e-65eb0d382584","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad6d6 0301ac6f-7f8e-4a41-a36e-65eb0d382584 /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0301ac6f-7f8e-4a41-a36e-65eb0d382584 {"id":"0301ac6f-7f8e-4a41-a36e-65eb0d382584","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad6d6 0301ac6f-7f8e-4a41-a36e-65eb0d382584 /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0301ac6f-7f8e-4a41-a36e-65eb0d382584 {"id":"0301ac6f-7f8e-4a41-a36e-65eb0d382584","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad6d6 0301ac6f-7f8e-4a41-a36e-65eb0d382584 /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.8228.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7489361070460815416,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:46:11.508379461+08:00","start_ts":457114326514532353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114326514532353,"checkpoint_ts":457114326514532353,"checkpoint_time":"2025-04-04 15:46:11.433"} PASS run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 table partition_table.finish_mark not exists for 7-th check, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 table batch_add_table.finish_mark not exists for 14-th check, retry later check diff failed 2-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed2?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed2.toml + log_suffix=changefeed2 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered ++ date + echo '[Fri Apr 4 15:46:12 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>>' [Fri Apr 4 15:46:12 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed2.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc_storage_consumerchangefeed2.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed2?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed2.toml table processor_delay.t14 exists table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 + set +x + workdir=/tmp/tidb_cdc_test/ddl_only_block_related_table + sink_uri='file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_only_block_related_table ++ date + echo '[Fri Apr 4 15:46:12 CST 2025] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>>' [Fri Apr 4 15:46:12 CST 2025] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>> + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table ddl_sequence.finish_mark not exists for 22-th check, retry later check diff failed 11-th time, retry later run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 table batch_add_table.finish_mark not exists for 15-th check, retry later table partition_table.finish_mark not exists for 8-th check, retry later run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t16 not exists for 2-th check, retry later check diff failed 3-th time, retry later table ddl_sequence.finish_mark not exists for 23-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later check diff failed 12-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 16-th check, retry later table partition_table.finish_mark not exists for 9-th check, retry later table processor_delay.t16 not exists for 3-th check, retry later check diff failed 4-th time, retry later table ddl_sequence.finish_mark not exists for 24-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 2-th time, retry later table batch_add_table.finish_mark not exists for 17-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later check diff failed 13-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later table processor_delay.t16 not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:46:16 CST 2025] <<<<<< run test case multi_source success! >>>>>> table ddl_sequence.finish_mark not exists for 25-th check, retry later check diff failed 5-th time, retry later table partition_table.finish_mark not exists for 11-th check, retry later check diff failed 14-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table batch_add_table.finish_mark not exists for 18-th check, retry later table processor_delay.t16 not exists for 5-th check, retry later table ddl_sequence.finish_mark exists check diff failed 1-th time, retry later check diff successfully check diff failed 6-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/vector/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 3-th time, retry later wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 12-th check, retry later check diff successfully table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark exists table test.t_1 does not exists check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:22 CST 2025] <<<<<< run test case ddl_sequence success! >>>>>> wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_with_exists/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:46:20 CST 2025] <<<<<< run test case ddl_with_exists success! >>>>>> check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... table processor_delay.t16 exists table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... check diff failed 7-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:23 CST 2025] <<<<<< run test case event_filter success! >>>>>> table ddl_only_block_related_table.finish_mark not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:23 CST 2025] <<<<<< run test case batch_add_table success! >>>>>> [2025/04/04 15:46:18.644 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:46:18.644 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489360374070982322] [2025/04/04 15:46:18.644 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:18.644 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:46:18.645 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:18.645 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:18.645 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:46:18.645 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:46:18.645 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:18.645 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:46:18.646 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:46:18.646 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489360374070982322] [2025/04/04 15:46:18.646 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:18.646 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:46:18.647 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:18.647 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:46:18.647 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:46:18.647 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:46:18.647 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:46:18.647 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:18.647 +08:00] [INFO] [tikv_driver.go:201] ["using API V1."] [2025/04/04 15:46:18.648 +08:00] [INFO] [main.go:180] ["genLock started"] [2025/04/04 15:46:18.649 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=3] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:20.648 +08:00] [INFO] [pd.go:441] ["adaptive update ts interval state transition"] [configuredInterval=2s] [prevAdaptiveUpdateInterval=2s] [newAdaptiveUpdateInterval=2s] [requiredStaleness=0s] [prevState=unknown(0)] [newState=normal] table partition_table.finish_mark not exists for 13-th check, retry later check diff failed 3-th time, retry later table processor_delay.t18 not exists for 2-th check, retry later check diff failed 8-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/vector Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } table ddl_only_block_related_table.finish_mark exists [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table partition_table.finish_mark not exists for 14-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } wait process 8164 exit for 1-th time... [Pipeline] // stage [Pipeline] } table processor_delay.t18 not exists for 3-th check, retry later wait process 8164 exit for 2-th time... check diff failed 4-th time, retry later /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8164) - No such process wait process 8164 exit for 3-th time... process 8164 already exit [Fri Apr 4 15:46:27 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.83788380.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 9-th time, retry later table partition_table.finish_mark not exists for 15-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 4-th time, retry later table processor_delay.t18 not exists for 4-th check, retry later check diff failed 5-th time, retry later [2025/04/04 15:46:28.648 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=148] [addr=127.0.0.1:9500] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:28.648 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=1] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:28.648 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=2] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:46:28.653 +08:00] [INFO] [main.go:196] ["genLock done"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_dispatcher.go:158] ["[tso] exit tso deadline watcher"] [dc-location=global] [2025/04/04 15:46:28.653 +08:00] [INFO] [resource_manager_client.go:296] ["[resource manager] exit resource token dispatcher"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_client.go:147] ["[tso] exit tso dispatcher check loop"] [2025/04/04 15:46:28.653 +08:00] [INFO] [pd_service_discovery.go:551] ["[pd] exit member loop due to context canceled"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_client.go:157] ["[tso] closing tso client"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_dispatcher.go:490] ["[tso] exit tso connection contexts updater"] [dc-location=global] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_dispatcher.go:264] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_dispatcher.go:201] ["[tso] exit tso dispatcher"] [dc-location=global] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_client.go:162] ["[tso] close tso client"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_stream.go:359] ["tsoStream.recvLoop ended"] [stream=127.0.0.1:2379-1] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241111073742-238d4d79ea31/tso_stream.go:427\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700"] [2025/04/04 15:46:28.653 +08:00] [INFO] [tso_client.go:164] ["[tso] tso client is closed"] [2025/04/04 15:46:28.653 +08:00] [INFO] [pd_service_discovery.go:644] ["[pd] close pd service discovery client"] check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 16-th check, retry later wait process cdc.test exit for 2-th time... table processor_delay.t18 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ca510 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331272183818 MinTableBarrierTs:457114331272183819 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331272183818, checkpointTs: 457114331272183818, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19ec0e1f-4b56-43ed-a675-af307e649152 {"id":"19ec0e1f-4b56-43ed-a675-af307e649152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752787} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad7c7 19ec0e1f-4b56-43ed-a675-af307e649152 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331272183818,"min-table-barrier-ts":457114331272183819,"admin-job-type":0} /tidb/cdc/default/default/task/position/19ec0e1f-4b56-43ed-a675-af307e649152/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ca510 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331272183818 MinTableBarrierTs:457114331272183819 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331272183818, checkpointTs: 457114331272183818, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19ec0e1f-4b56-43ed-a675-af307e649152 {"id":"19ec0e1f-4b56-43ed-a675-af307e649152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752787} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad7c7 19ec0e1f-4b56-43ed-a675-af307e649152 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331272183818,"min-table-barrier-ts":457114331272183819,"admin-job-type":0} /tidb/cdc/default/default/task/position/19ec0e1f-4b56-43ed-a675-af307e649152/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ca510 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331272183818 MinTableBarrierTs:457114331272183819 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331272183819, checkpointTs: 457114331272183819, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331272183818, checkpointTs: 457114331272183818, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19ec0e1f-4b56-43ed-a675-af307e649152 {"id":"19ec0e1f-4b56-43ed-a675-af307e649152","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752787} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad7c7 19ec0e1f-4b56-43ed-a675-af307e649152 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331272183818,"min-table-barrier-ts":457114331272183819,"admin-job-type":0} /tidb/cdc/default/default/task/position/19ec0e1f-4b56-43ed-a675-af307e649152/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:30 CST 2025] <<<<<< run test case changefeed_pause_resume success! >>>>>> check_ts_not_forward ddl-only-block-related-table Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 6-th time, retry later table partition_table.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table processor_delay.t18 exists table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:33 CST 2025] <<<<<< run test case csv_storage_partition_table success! >>>>>> table processor_delay.t20 not exists for 2-th check, retry later run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361193348440796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1032840007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:6893, start at 2025-04-04 15:46:33.260648848 +0800 CST m=+2.292569935 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.268 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361193348440796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1032840007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:6893, start at 2025-04-04 15:46:33.260648848 +0800 CST m=+2.292569935 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.268 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361186703197425 TiDB Cluster ID. tikv_gc_leader_uuid 657ff103408000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-p0q-b40s0, pid:6955, start at 2025-04-04 15:46:33.368442374 +0800 CST m=+2.311754460 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:33.376 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:33.346 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:33.346 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 8-th time, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/vector/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/vector/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/vector/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/vector/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/vector/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 run task successfully table processor_delay.t20 not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... table test.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... check_ts_not_forward ddl-only-block-related-table table test.t2 not exists for 2-th check, retry later check diff failed 9-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.8304.out cli tso query --pd=http://127.0.0.1:2379 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:37 CST 2025] <<<<<< run test case force_replicate_table success! >>>>>> table processor_delay.t20 not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x + tso='457114333400006657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114333400006657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:46:39 CST 2025] <<<<<< START cdc server in vector case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.83468348.out server --log-file /tmp/tidb_cdc_test/vector/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/vector/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 10-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.t2 not exists for 3-th check, retry later table processor_delay.t20 not exists for 5-th check, retry later Verifying downstream PD is started... Verifying downstream PD is started... check diff failed 11-th time, retry later table test.t2 not exists for 4-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:42 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f22c31a-3db5-4a1f-8423-2ad80c22855f {"id":"2f22c31a-3db5-4a1f-8423-2ad80c22855f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752799} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3edbee1 2f22c31a-3db5-4a1f-8423-2ad80c22855f /tidb/cdc/default/default/upstream/7489361193348440796 {"id":7489361193348440796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f22c31a-3db5-4a1f-8423-2ad80c22855f {"id":"2f22c31a-3db5-4a1f-8423-2ad80c22855f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752799} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3edbee1 2f22c31a-3db5-4a1f-8423-2ad80c22855f /tidb/cdc/default/default/upstream/7489361193348440796 {"id":7489361193348440796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f22c31a-3db5-4a1f-8423-2ad80c22855f {"id":"2f22c31a-3db5-4a1f-8423-2ad80c22855f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752799} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3edbee1 2f22c31a-3db5-4a1f-8423-2ad80c22855f /tidb/cdc/default/default/upstream/7489361193348440796 {"id":7489361193348440796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.8408.out cli changefeed create --start-ts=457114333400006657 '--sink-uri=file:///tmp/tidb_cdc_test/vector/storage_test/vector-19514?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 8e12d160-108f-4b62-90aa-120a10cd9fc6 Info: {"upstream_id":7489361193348440796,"namespace":"default","id":"8e12d160-108f-4b62-90aa-120a10cd9fc6","sink_uri":"file:///tmp/tidb_cdc_test/vector/storage_test/vector-19514?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:46:42.617000461+08:00","start_ts":457114333400006657,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114333400006657,"checkpoint_ts":457114333400006657,"checkpoint_time":"2025-04-04 15:46:37.699"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table processor_delay.t20 exists table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used table test.t2 not exists for 5-th check, retry later check diff failed 12-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/vector + sink_uri='file:///tmp/tidb_cdc_test/vector/storage_test/vector-19514?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/vector ++ date + echo '[Fri Apr 4 15:46:44 CST 2025] <<<<<< START storage consumer in vector case >>>>>>' [Fri Apr 4 15:46:44 CST 2025] <<<<<< START storage consumer in vector case >>>>>> + cd /tmp/tidb_cdc_test/vector + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/vector + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/vector/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/vector/storage_test/vector-19514?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t22 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 6-th check, retry later check diff failed 13-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t22 not exists for 3-th check, retry later Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m check diff failed 14-th time, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) table processor_delay.t22 not exists for 4-th check, retry later [Pipeline] // cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361255532307754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff110330000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9614, start at 2025-04-04 15:46:46.643864566 +0800 CST m=+2.395364729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.654 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.654 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361257120381775 TiDB Cluster ID. tikv_gc_leader_uuid 657ff11104c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:10407, start at 2025-04-04 15:46:47.464042931 +0800 CST m=+2.195342572 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:47.471 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:47.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:47.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361257120381775 TiDB Cluster ID. tikv_gc_leader_uuid 657ff11104c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:10407, start at 2025-04-04 15:46:47.464042931 +0800 CST m=+2.195342572 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:47.471 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:47.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:47.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361258449467553 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1111c4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-0s9-77gxr, pid:10473, start at 2025-04-04 15:46:47.580874229 +0800 CST m=+2.225492909 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:47.588 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:47.588 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:47.588 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 2-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 8-th check, retry later check diff failed 15-th time, retry later Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table processor_delay.t22 not exists for 5-th check, retry later [Pipeline] // node [Pipeline] } + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10861.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11787.out cli tso query --pd=http://127.0.0.1:2379 table test.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 9-th check, retry later check diff failed 16-th time, retry later + set +x + tso='457114336877346817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114336877346817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114336877346817 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:46:52 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1089710899.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore table processor_delay.t22 exists table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='457114337147092993 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114337147092993 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:46:53 CST 2025] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1182311825.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 10-th check, retry later check diff failed 17-th time, retry later table processor_delay.t24 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:55 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114336877346817 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10951.out cli changefeed create --start-ts=457114336877346817 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 table test.t2 not exists for 11-th check, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361252213972375,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:46:55.945738479+08:00","start_ts":457114336877346817,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114336877346817,"checkpoint_ts":457114336877346817,"checkpoint_time":"2025-04-04 15:46:50.964"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 18-th time, retry later table processor_delay.t24 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:56 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c18f50e-7b54-4a05-907f-56f13ccb49e2 {"id":"0c18f50e-7b54-4a05-907f-56f13ccb49e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752813} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42595e4 0c18f50e-7b54-4a05-907f-56f13ccb49e2 /tidb/cdc/default/default/upstream/7489361257120381775 {"id":7489361257120381775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c18f50e-7b54-4a05-907f-56f13ccb49e2 {"id":"0c18f50e-7b54-4a05-907f-56f13ccb49e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752813} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42595e4 0c18f50e-7b54-4a05-907f-56f13ccb49e2 /tidb/cdc/default/default/upstream/7489361257120381775 {"id":7489361257120381775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c18f50e-7b54-4a05-907f-56f13ccb49e2 {"id":"0c18f50e-7b54-4a05-907f-56f13ccb49e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752813} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42595e4 0c18f50e-7b54-4a05-907f-56f13ccb49e2 /tidb/cdc/default/default/upstream/7489361257120381775 {"id":7489361257120381775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11883.out cli changefeed create --start-ts=457114337147092993 '--sink-uri=file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-2908?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: b135607e-9dc9-4dec-9f87-323ee3af23b0 Info: {"upstream_id":7489361257120381775,"namespace":"default","id":"b135607e-9dc9-4dec-9f87-323ee3af23b0","sink_uri":"file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-2908?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:46:56.981204723+08:00","start_ts":457114337147092993,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114337147092993,"checkpoint_ts":457114337147092993,"checkpoint_time":"2025-04-04 15:46:51.993"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 table test.finish_mark not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361288666723911 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119d8c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:11002, start at 2025-04-04 15:46:56.525356149 +0800 CST m=+3.678505242 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.533 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361288666723911 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119d8c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:11002, start at 2025-04-04 15:46:56.525356149 +0800 CST m=+3.678505242 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.533 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361282199799631 TiDB Cluster ID. tikv_gc_leader_uuid 657ff119ad8000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:11056, start at 2025-04-04 15:46:56.32564766 +0800 CST m=+3.409957069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:56.333 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:56.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:56.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + '[' storage == mysql ']' + stop_tidb_cluster ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 12-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/resourcecontrol + sink_uri='file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-2908?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resourcecontrol ++ date + echo '[Fri Apr 4 15:46:58 CST 2025] <<<<<< START storage consumer in resourcecontrol case >>>>>>' [Fri Apr 4 15:46:58 CST 2025] <<<<<< START storage consumer in resourcecontrol case >>>>>> + cd /tmp/tidb_cdc_test/resourcecontrol + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resourcecontrol + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-2908?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_delay.t24 not exists for 4-th check, retry later table resourcecontrol.finish_mark not exists for 1-th check, retry later check diff failed 19-th time, retry later table test.finish_mark not exists for 7-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12424.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 13-th check, retry later table resourcecontrol.finish_mark not exists for 2-th check, retry later check diff failed 20-th time, retry later table processor_delay.t24 not exists for 5-th check, retry later table test.finish_mark not exists for 8-th check, retry later + set +x + tso='457114339202301953 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114339202301953 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 14-th check, retry later check diff failed 21-th time, retry later table processor_delay.t24 exists table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later table resourcecontrol.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 15-th check, retry later table processor_delay.t26 not exists for 2-th check, retry later table resourcecontrol.finish_mark not exists for 4-th check, retry later [Fri Apr 4 15:47:04 CST 2025] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1247712479.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 22-th time, retry later table test.finish_mark exists check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table test.t2 exists wait process cdc.test exit for 2-th time... table processor_delay.t26 not exists for 3-th check, retry later table resourcecontrol.finish_mark not exists for 5-th check, retry later check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:07 CST 2025] <<<<<< run test case vector success! >>>>>> check diff failed 23-th time, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:07 GMT < Content-Length: 862 < Content-Type: text/plain; charset=utf-8 < { [862 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ef0cc1d-3363-49c3-a7e6-82343065659b {"id":"1ef0cc1d-3363-49c3-a7e6-82343065659b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc446ed61 1ef0cc1d-3363-49c3-a7e6-82343065659b /tidb/cdc/default/default/upstream/7489361288666723911 {"id":7489361288666723911,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ef0cc1d-3363-49c3-a7e6-82343065659b {"id":"1ef0cc1d-3363-49c3-a7e6-82343065659b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc446ed61 1ef0cc1d-3363-49c3-a7e6-82343065659b /tidb/cdc/default/default/upstream/7489361288666723911 {"id":7489361288666723911,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ef0cc1d-3363-49c3-a7e6-82343065659b {"id":"1ef0cc1d-3363-49c3-a7e6-82343065659b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752825} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc446ed61 1ef0cc1d-3363-49c3-a7e6-82343065659b /tidb/cdc/default/default/upstream/7489361288666723911 {"id":7489361288666723911,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12542.out cli changefeed create --start-ts=457114339202301953 '--sink-uri=file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name wait process cdc.test exit for 2-th time... [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361288666723911,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:47:08.196078046+08:00","start_ts":457114339202301953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114339202301953,"checkpoint_ts":457114339202301953,"checkpoint_time":"2025-04-04 15:46:59.833"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table resourcecontrol.finish_mark exists check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:08 CST 2025] <<<<<< run test case resolve_lock success! >>>>>> check diff failed 24-th time, retry later table processor_delay.t26 not exists for 4-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + set +x + workdir=/tmp/tidb_cdc_test/cli_with_auth + sink_uri='file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/cli_with_auth ++ date + echo '[Fri Apr 4 15:47:09 CST 2025] <<<<<< START storage consumer in cli_with_auth case >>>>>>' [Fri Apr 4 15:47:09 CST 2025] <<<<<< START storage consumer in cli_with_auth case >>>>>> + cd /tmp/tidb_cdc_test/cli_with_auth + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/cli_with_auth + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:10 CST 2025] <<<<<< run test case resourcecontrol success! >>>>>> table processor_delay.t26 not exists for 5-th check, retry later check diff failed 25-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361249616670307 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1275a8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:7079, start at 2025-04-04 15:47:10.348370755 +0800 CST m=+24.863914304 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:10.356 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:10.314 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:10.314 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361249616670307 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1275a8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:7079, start at 2025-04-04 15:47:10.348370755 +0800 CST m=+24.863914304 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:10.356 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:10.314 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:10.314 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361257452641769 TiDB Cluster ID. tikv_gc_leader_uuid 657ff112d840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-lrn-bd4rs, pid:7144, start at 2025-04-04 15:46:49.314234525 +0800 CST m=+3.746525536 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:49.320 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:49.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:49.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.simple not exists for 2-th check, retry later + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... table processor_delay.t26 exists table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later check diff failed 26-th time, retry later table test.simple not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Fri Apr 4 15:47:14 CST 2025] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.2103621038.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t28 not exists for 2-th check, retry later check diff failed 27-th time, retry later table test.simple not exists for 4-th check, retry later table processor_delay.t28 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:17 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75db1d6c-9cb4-420e-83ed-39b9a9150f54 {"id":"75db1d6c-9cb4-420e-83ed-39b9a9150f54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752834} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4244c23 75db1d6c-9cb4-420e-83ed-39b9a9150f54 /tidb/cdc/default/default/upstream/7489361249616670307 {"id":7489361249616670307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75db1d6c-9cb4-420e-83ed-39b9a9150f54 {"id":"75db1d6c-9cb4-420e-83ed-39b9a9150f54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752834} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4244c23 75db1d6c-9cb4-420e-83ed-39b9a9150f54 /tidb/cdc/default/default/upstream/7489361249616670307 {"id":7489361249616670307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75db1d6c-9cb4-420e-83ed-39b9a9150f54 {"id":"75db1d6c-9cb4-420e-83ed-39b9a9150f54","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752834} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4244c23 75db1d6c-9cb4-420e-83ed-39b9a9150f54 /tidb/cdc/default/default/upstream/7489361249616670307 {"id":7489361249616670307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.21316.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-6987?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: cca65b77-8471-480c-9bc3-88104592506f Info: {"upstream_id":7489361249616670307,"namespace":"default","id":"cca65b77-8471-480c-9bc3-88104592506f","sink_uri":"file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-6987?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:47:17.504957767+08:00","start_ts":457114343811055618,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114343811055618,"checkpoint_ts":457114343811055618,"checkpoint_time":"2025-04-04 15:47:17.414"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 28-th time, retry later table test.simple not exists for 5-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/multi_rocks + sink_uri='file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-6987?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_rocks ++ date + echo '[Fri Apr 4 15:47:18 CST 2025] <<<<<< START storage consumer in multi_rocks case >>>>>>' [Fri Apr 4 15:47:18 CST 2025] <<<<<< START storage consumer in multi_rocks case >>>>>> + cd /tmp/tidb_cdc_test/multi_rocks + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_rocks + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_rocks/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-6987?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_delay.t28 not exists for 4-th check, retry later check diff failed 29-th time, retry later table test.simple not exists for 6-th check, retry later ***************** properties ***************** "mysql.user"="root" "readproportion"="0" "insertproportion"="0" "scanproportion"="0" "threadcount"="2" "updateproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.port"="4000" "table"="a1" "dotransactions"="false" "mysql.db"="multi_rocks" "workload"="core" "recordcount"="1000" "operationcount"="0" "readallfields"="true" ********************************************** Run finished, takes 593.54197ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1737.2, Avg(us): 1153, Min(us): 822, Max(us): 17863, 95th(us): 2000, 99th(us): 2000 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t28 not exists for 5-th check, retry later ***************** properties ***************** "table"="a2" "readallfields"="true" "operationcount"="0" "mysql.port"="4000" "requestdistribution"="uniform" "scanproportion"="0" "threadcount"="2" "updateproportion"="0" "mysql.user"="root" "readproportion"="0" "mysql.db"="multi_rocks" "recordcount"="1000" "mysql.host"="127.0.0.1" "insertproportion"="0" "workload"="core" "dotransactions"="false" ********************************************** Run finished, takes 599.893501ms INSERT - Takes(s): 0.6, Count: 999, OPS: 1726.7, Avg(us): 1148, Min(us): 868, Max(us): 21225, 95th(us): 2000, 99th(us): 2000 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } ***************** properties ***************** "insertproportion"="0" "requestdistribution"="uniform" "recordcount"="1000" "mysql.db"="multi_rocks" "dotransactions"="false" "mysql.user"="root" "table"="a3" "readproportion"="0" "workload"="core" "readallfields"="true" "threadcount"="2" "mysql.port"="4000" "scanproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "updateproportion"="0" ********************************************** [Pipeline] // timeout check diff failed 30-th time, retry later [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } table test.simple exists table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114344760016898, "checkpoint_time": "2025-04-04 15:47:21.034", "error": null }' + echo '{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114344760016898, "checkpoint_time": "2025-04-04 15:47:21.034", "error": null }' { "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114344760016898, "checkpoint_time": "2025-04-04 15:47:21.034", "error": null } ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114344760016898, '"checkpoint_time":' '"2025-04-04' '15:47:21.034",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114344760016898, '"checkpoint_time":' '"2025-04-04' '15:47:21.034",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 [Pipeline] // withEnv Run finished, takes 564.766069ms INSERT - Takes(s): 0.5, Count: 1000, OPS: 1834.5, Avg(us): 1100, Min(us): 815, Max(us): 19532, 95th(us): 2000, 99th(us): 2000 [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t28 exists table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "updateproportion"="0" "insertproportion"="0" "mysql.db"="multi_rocks" "operationcount"="0" "requestdistribution"="uniform" "recordcount"="1000" "mysql.user"="root" "readallfields"="true" "mysql.port"="4000" "workload"="core" "mysql.host"="127.0.0.1" "scanproportion"="0" "threadcount"="2" "dotransactions"="false" "readproportion"="0" "table"="a4" ********************************************** Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0038b1ba0} {Type:update Path:[Config SyncPointRetention] From: To:0xc0038b1ba8} {Type:update Path:[Config Consistent] From: To:0xc0009d8f50} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12852.out cli changefeed --changefeed-id custom-changefeed-name pause pass check, checkpoint tso not forward after 10s run task successfully start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed at last An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_uk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log for full details An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_pk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log for full details A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 A total of 2 table have been compared and all are equal. You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log' An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_pk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log for full details An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_uk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log for full details A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/0 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_pk` ... failure Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' cat: /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff/output/sync_diff.log: No such file or directory check_sync_diff failed as expected for changefeed2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.cli.13755.out cli changefeed create --start-ts=457114322070667269 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed3?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed3.toml -c changefeed3 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... Run finished, takes 742.404882ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1385.6, Avg(us): 1451, Min(us): 781, Max(us): 72321, 95th(us): 2000, 99th(us): 3000 [2025/04/04 15:47:24.244 +08:00] [WARN] [sink.go:900] ["TiCDC will not split the update pk/uk events if output-raw-change-event is true(scheme: file).It is not recommended to set output-old-value to false(protocol: csv) in this case.Otherwise, there may be data consistency issues in update pk/uk scenarios due to lack of old value information."] Create changefeed successfully! ID: changefeed3 Info: {"upstream_id":7489360998406299861,"namespace":"default","id":"changefeed3","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed3?flush-interval=5s","create_time":"2025-04-04T15:47:24.282965869+08:00","start_ts":457114322070667269,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":true},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114322070667269,"checkpoint_ts":457114322070667269,"checkpoint_time":"2025-04-04 15:45:54.481"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process 8383 exit for 1-th time... ***************** properties ***************** "dotransactions"="false" "readproportion"="0" "mysql.user"="root" "operationcount"="0" "workload"="core" "threadcount"="2" "updateproportion"="0" "recordcount"="1000" "readallfields"="true" "mysql.port"="4000" "table"="a5" "requestdistribution"="uniform" "mysql.db"="multi_rocks" "mysql.host"="127.0.0.1" "scanproportion"="0" "insertproportion"="0" ********************************************** table processor_delay.t30 not exists for 2-th check, retry later wait process 8383 exit for 2-th time... \033[0;36m<<< Run all test success >>>\033[0m VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361255532307754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff110330000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9614, start at 2025-04-04 15:46:46.643864566 +0800 CST m=+2.395364729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.654 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.654 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } Run finished, takes 568.024133ms INSERT - Takes(s): 0.5, Count: 1000, OPS: 1820.8, Avg(us): 1103, Min(us): 782, Max(us): 18690, 95th(us): 2000, 99th(us): 2000 [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials + set +x [Pipeline] } [Pipeline] // timeout [Pipeline] } wait process 8383 exit for 3-th time... + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed3?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed3.toml + log_suffix=changefeed3 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered ++ date + echo '[Fri Apr 4 15:47:25 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>>' [Fri Apr 4 15:47:25 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed3.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc_storage_consumerchangefeed3.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed3?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed3.toml [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } wait process 8383 exit for 4-th time... [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... table processor_delay.t30 not exists for 3-th check, retry later table multi_rocks.finish_mark not exists for 1-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8383) - No such process wait process 8383 exit for 5-th time... process 8383 already exit [Fri Apr 4 15:47:26 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.89838985.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114345284042756, "checkpoint_time": "2025-04-04 15:47:23.033", "error": null }' + echo '{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114345284042756, "checkpoint_time": "2025-04-04 15:47:23.033", "error": null }' { "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114345284042756, "checkpoint_time": "2025-04-04 15:47:23.033", "error": null } ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114345284042756, '"checkpoint_time":' '"2025-04-04' '15:47:23.033",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114345284042756, '"checkpoint_time":' '"2025-04-04' '15:47:23.033",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12913.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name table processor_delay.t30 not exists for 4-th check, retry later table multi_rocks.finish_mark not exists for 2-th check, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc003b699d8} {Type:update Path:[Config SyncPointRetention] From: To:0xc003b699e8} {Type:update Path:[Config Consistent] From: To:0xc000fac2a0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361288666723911,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:47:08.196078046+08:00","start_ts":457114339202301953,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114345284042756,"checkpoint_time":"2025-04-04 15:47:23.033"} PASS coverage: 2.8% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:29 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00456c480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331455684638 MinTableBarrierTs:457114346764894213 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54339343-bfa5-4139-b1ed-43abbc1b62d7 {"id":"54339343-bfa5-4139-b1ed-43abbc1b62d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad9b5 54339343-bfa5-4139-b1ed-43abbc1b62d7 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331455684638,"min-table-barrier-ts":457114346764894213,"admin-job-type":0} /tidb/cdc/default/default/task/position/54339343-bfa5-4139-b1ed-43abbc1b62d7/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00456c480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331455684638 MinTableBarrierTs:457114346764894213 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54339343-bfa5-4139-b1ed-43abbc1b62d7 {"id":"54339343-bfa5-4139-b1ed-43abbc1b62d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad9b5 54339343-bfa5-4139-b1ed-43abbc1b62d7 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331455684638,"min-table-barrier-ts":457114346764894213,"admin-job-type":0} /tidb/cdc/default/default/task/position/54339343-bfa5-4139-b1ed-43abbc1b62d7/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489361070460815416 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:46:11.508379461 +0800 CST StartTs:457114326514532353 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00456c480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114326527639557} {CheckpointTs:457114331455684638 MinTableBarrierTs:457114346764894213 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114331455684638, checkpointTs: 457114331455684638, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54339343-bfa5-4139-b1ed-43abbc1b62d7 {"id":"54339343-bfa5-4139-b1ed-43abbc1b62d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752846} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc37ad9b5 54339343-bfa5-4139-b1ed-43abbc1b62d7 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489361070460815416,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-6062?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:46:11.508379461+08:00","start-ts":457114326514532353,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114326527639557} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114331455684638,"min-table-barrier-ts":457114346764894213,"admin-job-type":0} /tidb/cdc/default/default/task/position/54339343-bfa5-4139-b1ed-43abbc1b62d7/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'etcd info' /tidb/cdc/default/default/upstream/7489361070460815416 {"id":7489361070460815416,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check diff failed 1-th time, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12945.out cli changefeed --changefeed-id custom-changefeed-name resume table processor_delay.t30 not exists for 5-th check, retry later table multi_rocks.finish_mark not exists for 3-th check, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table processor_delay.t30 exists table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later table multi_rocks.finish_mark not exists for 4-th check, retry later check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully check_ts_forward ddl-only-block-related-table table processor_delay.t32 not exists for 2-th check, retry later table multi_rocks.finish_mark not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114348430032900, "checkpoint_time": "2025-04-04 15:47:35.034", "error": null }' + echo '{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114348430032900, "checkpoint_time": "2025-04-04 15:47:35.034", "error": null }' { "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114348430032900, "checkpoint_time": "2025-04-04 15:47:35.034", "error": null } ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114348430032900, '"checkpoint_time":' '"2025-04-04' '15:47:35.034",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114348430032900, '"checkpoint_time":' '"2025-04-04' '15:47:35.034",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13030.out cli changefeed --changefeed-id custom-changefeed-name remove changefeed is working normally rts: 457114348075876355->457114348338020355 checkpoint: 457114348075876355->457114348338020355 run task successfully check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114348692176900 SinkURI: file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6906 0 --:--:-- --:--:-- --:--:-- 6906 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:47:24.173","puller_resolved_ts":"2025-04-04 15:47:20.173","last_synced_ts":"2025-04-04 15:45:08.673","now_ts":"2025-04-04 15:47:25.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:24.173","puller_resolved_ts":"2025-04-04' '15:47:20.173","last_synced_ts":"2025-04-04' '15:45:08.673","now_ts":"2025-04-04' '15:47:25.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 9340 10.0 0.0 15706348 148632 ? Sl 15:44 0:15 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 9403 7.2 0.0 15705580 143028 ? Sl 15:44 0:11 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1743752852.2756782,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0050c3c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752852.2757564,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752852.2970343,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0030e9340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752852.2971103,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752852.7185402,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0054c7a40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752852.7186103,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table processor_delay.t32 not exists for 3-th check, retry later table multi_rocks.finish_mark not exists for 6-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:37 CST 2025] <<<<<< run test case ddl_only_block_related_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later + set +x table processor_delay.t32 not exists for 4-th check, retry later table multi_rocks.finish_mark not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release {"level":"warn","ts":"2025-04-04T15:47:40.016953+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:40.02226+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:40.124903+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table processor_delay.t32 not exists for 5-th check, retry later table multi_rocks.finish_mark not exists for 8-th check, retry later changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13086.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489361288666723911,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-31112?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:47:41.986351809+08:00","start_ts":457114350238826497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114350238826497,"checkpoint_ts":457114350238826497,"checkpoint_time":"2025-04-04 15:47:41.934"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later table processor_delay.t32 exists table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later table multi_rocks.finish_mark not exists for 9-th check, retry later + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 6-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361255532307754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff110330000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9614, start at 2025-04-04 15:46:46.643864566 +0800 CST m=+2.395364729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.654 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.654 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... table processor_delay.t34 not exists for 2-th check, retry later table multi_rocks.finish_mark not exists for 10-th check, retry later {"level":"warn","ts":1743752865.362989,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0050c3c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752865.8043141,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0054c7a40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:46.01837+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:46.023867+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:46.126531+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114351313354756, "checkpoint_time": "2025-04-04 15:47:46.033", "error": null }' + echo '{ "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114351313354756, "checkpoint_time": "2025-04-04 15:47:46.033", "error": null }' { "upstream_id": 7489361288666723911, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114351313354756, "checkpoint_time": "2025-04-04 15:47:46.033", "error": null } ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114351313354756, '"checkpoint_time":' '"2025-04-04' '15:47:46.033",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361288666723911, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114351313354756, '"checkpoint_time":' '"2025-04-04' '15:47:46.033",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] check diff failed 7-th time, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13175.out cli unsafe delete-service-gc-safepoint table processor_delay.t34 not exists for 3-th check, retry later table multi_rocks.finish_mark not exists for 11-th check, retry later Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 8-th time, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13209.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 table processor_delay.t34 not exists for 4-th check, retry later table multi_rocks.finish_mark not exists for 12-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: storage... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 check diff failed 9-th time, retry later + set +x table processor_delay.t34 not exists for 5-th check, retry later table multi_rocks.finish_mark exists check diff successfully 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:52.019967+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:52.025863+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:52.127672+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:52 CST 2025] <<<<<< run test case multi_rocks success! >>>>>> table processor_delay.t34 exists table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later check diff failed 10-th time, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13295.out cli unsafe resolve-lock --region=30 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361443945674574 TiDB Cluster ID. tikv_gc_leader_uuid 657ff151e3c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:23895, start at 2025-04-04 15:47:53.873794771 +0800 CST m=+24.138366867 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:53.880 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:53.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:53.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t36 not exists for 2-th check, retry later check diff failed 11-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13327.out cli unsafe resolve-lock --region=30 --ts=457114352585015297 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361443945674574 TiDB Cluster ID. tikv_gc_leader_uuid 657ff151e3c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:23895, start at 2025-04-04 15:47:53.873794771 +0800 CST m=+24.138366867 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:53.880 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:53.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:53.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361447494026002 TiDB Cluster ID. tikv_gc_leader_uuid 657ff13c030000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:23971, start at 2025-04-04 15:47:31.518157807 +0800 CST m=+1.695209659 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:31.524 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:31.518 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:31.518 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 table processor_delay.t36 not exists for 3-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 207k 6000 --:--:-- --:--:-- --:--:-- 212k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 12-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:58.021113+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:58.027158+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:58.129003+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.25332.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t36 not exists for 4-th check, retry later check diff failed 13-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:00.008494+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:48:00.008573+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:48:00.011843+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:48:00.011888+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='457114354600378369 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114354600378369 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "mysql.host"="127.0.0.1" "readallfields"="true" "mysql.user"="root" "dotransactions"="false" "readproportion"="0" "operationcount"="0" "updateproportion"="0" "requestdistribution"="uniform" "workload"="core" "threadcount"="10" "mysql.port"="4000" "scanproportion"="0" "mysql.db"="move_table" "recordcount"="10000" "insertproportion"="0" ********************************************** 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:00.112188+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:48:00.112247+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 178k 0 --:--:-- --:--:-- --:--:-- 178k { "version": "v9.0.0-beta.1-10-ge8bfbe4ba", "git_hash": "e8bfbe4ba98e05140ab733ee3d08dd3b99918721", "id": "31237317-c5eb-4db3-ae28-b26158ca9a5f", "pid": 12482, "is_owner": true }wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t36 not exists for 5-th check, retry later check diff failed 14-th time, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:02 CST 2025] <<<<<< run test case cli_with_auth success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Run finished, takes 2.428551037s INSERT - Takes(s): 2.4, Count: 10000, OPS: 4126.2, Avg(us): 2383, Min(us): 836, Max(us): 61818, 95th(us): 3000, 99th(us): 41000 [Fri Apr 4 15:48:02 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2541125413.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t36 exists table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361573447197853 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1597dc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:10239, start at 2025-04-04 15:48:01.68309418 +0800 CST m=+2.115826405 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:01.732 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:01.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:01.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361573447197853 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1597dc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:10239, start at 2025-04-04 15:48:01.68309418 +0800 CST m=+2.115826405 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:01.732 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:01.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:01.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361578692881097 TiDB Cluster ID. tikv_gc_leader_uuid 657ff159a380004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-nb3-whwdx, pid:10305, start at 2025-04-04 15:48:01.835946917 +0800 CST m=+2.185110559 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:01.845 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:01.806 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:01.806 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... check diff failed 15-th time, retry later TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:04.02252+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:48:04.02863+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:04.130126+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ci_collation_compatibility/run.sh using Sink-Type: storage... <<================= [Fri Apr 4 15:48:04 CST 2025] <<<<<< run test case ci_collation_compatibility success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361252213972375 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1103000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9534, start at 2025-04-04 15:46:46.608890533 +0800 CST m=+2.436133586 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361255532307754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff110330000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5zf-n347m, pid:9614, start at 2025-04-04 15:46:46.643864566 +0800 CST m=+2.395364729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:46.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:46.654 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:46.654 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14709.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t38 not exists for 2-th check, retry later check diff failed 16-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:05 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.25465.out cli changefeed create --start-ts=457114354600378369 '--sink-uri=file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 05196a3b-b86d-44be-a5d8-68612b795c06 Info: {"upstream_id":7489361443945674574,"namespace":"default","id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink_uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:48:06.086241165+08:00","start_ts":457114354600378369,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114354600378369,"checkpoint_ts":457114354600378369,"checkpoint_time":"2025-04-04 15:47:58.572"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ***************** properties ***************** "requestdistribution"="uniform" "mysql.user"="root" "updateproportion"="0" "threadcount"="4" "operationcount"="0" "dotransactions"="false" "mysql.db"="changefeed_auto_stop_1" "insertproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="20" "scanproportion"="0" "mysql.port"="4000" "workload"="core" "readallfields"="true" "readproportion"="0" ********************************************** Run finished, takes 9.212662ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4075.5, Avg(us): 1725, Min(us): 913, Max(us): 4276, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "readallfields"="true" "scanproportion"="0" "threadcount"="4" "readproportion"="0" "recordcount"="20" "mysql.user"="root" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.port"="4000" "insertproportion"="0" "dotransactions"="false" "workload"="core" "operationcount"="0" "updateproportion"="0" "mysql.db"="changefeed_auto_stop_2" ********************************************** Run finished, takes 9.745119ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3282.0, Avg(us): 1800, Min(us): 1054, Max(us): 3892, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "updateproportion"="0" "insertproportion"="0" "mysql.port"="4000" "mysql.host"="127.0.0.1" "operationcount"="0" "workload"="core" "mysql.db"="changefeed_auto_stop_3" "readproportion"="0" "mysql.user"="root" "scanproportion"="0" "dotransactions"="false" "readallfields"="true" "requestdistribution"="uniform" "threadcount"="4" "recordcount"="20" ********************************************** Run finished, takes 8.613089ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4129.9, Avg(us): 1604, Min(us): 928, Max(us): 3759, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "readallfields"="true" "dotransactions"="false" "scanproportion"="0" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.user"="root" "threadcount"="4" "workload"="core" "mysql.host"="127.0.0.1" "readproportion"="0" "updateproportion"="0" "mysql.db"="changefeed_auto_stop_4" "recordcount"="20" "insertproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 9.353386ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3604.8, Avg(us): 1739, Min(us): 1011, Max(us): 4000, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:48:06 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1172811730.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114356343635969 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114356343635969 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114356343635969 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:48:06 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1474314745.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 table processor_delay.t38 not exists for 3-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1743752887.277669,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0050c3c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752887.2777393,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752887.298439,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0030e9340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752887.2985158,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} + set +x + workdir=/tmp/tidb_cdc_test/move_table + sink_uri='file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/move_table ++ date + echo '[Fri Apr 4 15:48:07 CST 2025] <<<<<< START storage consumer in move_table case >>>>>>' [Fri Apr 4 15:48:07 CST 2025] <<<<<< START storage consumer in move_table case >>>>>> + cd /tmp/tidb_cdc_test/move_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/move_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/move_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Fri Apr 4 15:48:07 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2550425510.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL check diff failed 17-th time, retry later {"level":"warn","ts":1743752887.7196834,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0054c7a40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752887.7197714,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t38 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:09 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:48:09 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1178611788.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:10.023824+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:48:10.030259+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:48:10.13177+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} check diff failed 18-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:09 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489361252213972375 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:55.945738479 +0800 CST StartTs:457114336877346817 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393a900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114338174959622} {CheckpointTs:457114338515746825 MinTableBarrierTs:457114338515746825 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1043cb6c-566d-41e2-9b5b-9adbaeb6635f {"id":"1043cb6c-566d-41e2-9b5b-9adbaeb6635f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/__cdc_meta__/owner/223195ffc54948d3 1043cb6c-566d-41e2-9b5b-9adbaeb6635f /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489361252213972375,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:55.945738479+08:00","start-ts":457114336877346817,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114338174959622} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114338515746825,"min-table-barrier-ts":457114338515746825,"admin-job-type":0} /tidb/cdc/default/default/task/position/1043cb6c-566d-41e2-9b5b-9adbaeb6635f/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bf49a81e-bdd9-4065-96a6-7a3999508bba/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489361252213972375 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:55.945738479 +0800 CST StartTs:457114336877346817 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393a900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114338174959622} {CheckpointTs:457114338515746825 MinTableBarrierTs:457114338515746825 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1043cb6c-566d-41e2-9b5b-9adbaeb6635f {"id":"1043cb6c-566d-41e2-9b5b-9adbaeb6635f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/__cdc_meta__/owner/223195ffc54948d3 1043cb6c-566d-41e2-9b5b-9adbaeb6635f /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489361252213972375,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:55.945738479+08:00","start-ts":457114336877346817,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114338174959622} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114338515746825,"min-table-barrier-ts":457114338515746825,"admin-job-type":0} /tidb/cdc/default/default/task/position/1043cb6c-566d-41e2-9b5b-9adbaeb6635f/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bf49a81e-bdd9-4065-96a6-7a3999508bba/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489361252213972375 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:46:55.945738479 +0800 CST StartTs:457114336877346817 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393a900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114338174959622} {CheckpointTs:457114338515746825 MinTableBarrierTs:457114338515746825 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1043cb6c-566d-41e2-9b5b-9adbaeb6635f {"id":"1043cb6c-566d-41e2-9b5b-9adbaeb6635f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/bf49a81e-bdd9-4065-96a6-7a3999508bba {"id":"bf49a81e-bdd9-4065-96a6-7a3999508bba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752812} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc42316ec bf49a81e-bdd9-4065-96a6-7a3999508bba /tidb/cdc/default/__cdc_meta__/owner/223195ffc54948d3 1043cb6c-566d-41e2-9b5b-9adbaeb6635f /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489361252213972375,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:46:55.945738479+08:00","start-ts":457114336877346817,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114338174959622} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114338515746825,"min-table-barrier-ts":457114338515746825,"admin-job-type":0} /tidb/cdc/default/default/task/position/1043cb6c-566d-41e2-9b5b-9adbaeb6635f/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bf49a81e-bdd9-4065-96a6-7a3999508bba/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361252213972375 {"id":7489361252213972375,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114356343635969 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14802.out cli changefeed create --start-ts=457114356343635969 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:10 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016662d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114354993332238 MinTableBarrierTs:457114357549498376 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114354993332238,"min-table-barrier-ts":457114357549498376,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016662d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114354993332238 MinTableBarrierTs:457114357549498376 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114354993332238,"min-table-barrier-ts":457114357549498376,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016662d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114354993332238 MinTableBarrierTs:457114357549498376 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114354993332238,"min-table-barrier-ts":457114357549498376,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:48:10 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2560425606.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t38 not exists for 5-th check, retry later check diff failed 19-th time, retry later Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489361252213972375,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:48:12.428101459+08:00","start_ts":457114356343635969,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114356343635969,"checkpoint_ts":457114356343635969,"checkpoint_time":"2025-04-04 15:48:05.222"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table processor_delay.t38 exists table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:12 GMT < Content-Length: 1277 < Content-Type: text/plain; charset=utf-8 < { [1277 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/capture/78429c8c-5ba0-4b9e-9512-5800ddd0125d {"id":"78429c8c-5ba0-4b9e-9512-5800ddd0125d","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752889} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a055 78429c8c-5ba0-4b9e-9512-5800ddd0125d /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/capture/78429c8c-5ba0-4b9e-9512-5800ddd0125d {"id":"78429c8c-5ba0-4b9e-9512-5800ddd0125d","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752889} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a055 78429c8c-5ba0-4b9e-9512-5800ddd0125d /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0568664e-55e9-4c80-9d63-996e7b0fc1cb {"id":"0568664e-55e9-4c80-9d63-996e7b0fc1cb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752886} /tidb/cdc/default/__cdc_meta__/capture/78429c8c-5ba0-4b9e-9512-5800ddd0125d {"id":"78429c8c-5ba0-4b9e-9512-5800ddd0125d","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752889} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a018 0568664e-55e9-4c80-9d63-996e7b0fc1cb /tidb/cdc/default/__cdc_meta__/owner/223195ffc549a055 78429c8c-5ba0-4b9e-9512-5800ddd0125d /tidb/cdc/default/default/upstream/7489361573447197853 {"id":7489361573447197853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_auto_stop + sink_uri='file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-9370?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_auto_stop ++ date + echo '[Fri Apr 4 15:48:12 CST 2025] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>>' [Fri Apr 4 15:48:12 CST 2025] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_auto_stop + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_auto_stop + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-9370?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 8fb8a7d2-9899-4809-b890-9888bd5f2590 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=8fb8a7d2-9899-4809-b890-9888bd5f2590 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 8fb8a7d2-9899-4809-b890-9888bd5f2590 -s + info='{ "upstream_id": 7489361573447197853, "namespace": "default", "id": "8fb8a7d2-9899-4809-b890-9888bd5f2590", "state": "normal", "checkpoint_tso": 457114356548632577, "checkpoint_time": "2025-04-04 15:48:06.004", "error": null }' + echo '{ "upstream_id": 7489361573447197853, "namespace": "default", "id": "8fb8a7d2-9899-4809-b890-9888bd5f2590", "state": "normal", "checkpoint_tso": 457114356548632577, "checkpoint_time": "2025-04-04 15:48:06.004", "error": null }' { "upstream_id": 7489361573447197853, "namespace": "default", "id": "8fb8a7d2-9899-4809-b890-9888bd5f2590", "state": "normal", "checkpoint_tso": 457114356548632577, "checkpoint_time": "2025-04-04 15:48:06.004", "error": null } ++ echo '{' '"upstream_id":' 7489361573447197853, '"namespace":' '"default",' '"id":' '"8fb8a7d2-9899-4809-b890-9888bd5f2590",' '"state":' '"normal",' '"checkpoint_tso":' 457114356548632577, '"checkpoint_time":' '"2025-04-04' '15:48:06.004",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489361573447197853, '"namespace":' '"default",' '"id":' '"8fb8a7d2-9899-4809-b890-9888bd5f2590",' '"state":' '"normal",' '"checkpoint_tso":' 457114356548632577, '"checkpoint_time":' '"2025-04-04' '15:48:06.004",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042b4240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114355163725863 MinTableBarrierTs:457114358354280458 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/f024d26e-1a51-4645-b7fd-40c0849e0bbd {"id":"f024d26e-1a51-4645-b7fd-40c0849e0bbd","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752891} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321f4 f024d26e-1a51-4645-b7fd-40c0849e0bbd /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114355163725863,"min-table-barrier-ts":457114358354280458,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f024d26e-1a51-4645-b7fd-40c0849e0bbd/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042b4240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114355163725863 MinTableBarrierTs:457114358354280458 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/f024d26e-1a51-4645-b7fd-40c0849e0bbd {"id":"f024d26e-1a51-4645-b7fd-40c0849e0bbd","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752891} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321f4 f024d26e-1a51-4645-b7fd-40c0849e0bbd /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114355163725863,"min-table-barrier-ts":457114358354280458,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f024d26e-1a51-4645-b7fd-40c0849e0bbd/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/05196a3b-b86d-44be-a5d8-68612b795c06 {UpstreamID:7489361443945674574 Namespace:default ID:05196a3b-b86d-44be-a5d8-68612b795c06 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:48:06.086241165 +0800 CST StartTs:457114354600378369 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042b4240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114356566458372} {CheckpointTs:457114355163725863 MinTableBarrierTs:457114358354280458 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a24660d-444f-4ade-854a-f72fd2e294e2 {"id":"2a24660d-444f-4ade-854a-f72fd2e294e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752882} /tidb/cdc/default/__cdc_meta__/capture/677a764e-135c-489f-95c4-aa6441131043 {"id":"677a764e-135c-489f-95c4-aa6441131043","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752887} /tidb/cdc/default/__cdc_meta__/capture/f024d26e-1a51-4645-b7fd-40c0849e0bbd {"id":"f024d26e-1a51-4645-b7fd-40c0849e0bbd","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752891} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d32166 2a24660d-444f-4ade-854a-f72fd2e294e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321bd 677a764e-135c-489f-95c4-aa6441131043 /tidb/cdc/default/__cdc_meta__/owner/223195ffc4d321f4 f024d26e-1a51-4645-b7fd-40c0849e0bbd /tidb/cdc/default/default/changefeed/info/05196a3b-b86d-44be-a5d8-68612b795c06 {"upstream-id":7489361443945674574,"namespace":"default","changefeed-id":"05196a3b-b86d-44be-a5d8-68612b795c06","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-14381?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:48:06.086241165+08:00","start-ts":457114354600378369,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114356566458372} /tidb/cdc/default/default/changefeed/status/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":457114355163725863,"min-table-barrier-ts":457114358354280458,"admin-job-type":0} /tidb/cdc/default/default/task/position/2a24660d-444f-4ade-854a-f72fd2e294e2/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/677a764e-135c-489f-95c4-aa6441131043/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f024d26e-1a51-4645-b7fd-40c0849e0bbd/05196a3b-b86d-44be-a5d8-68612b795c06 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489361443945674574 {"id":7489361443945674574,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table move_table.usertable not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 20-th time, retry later table changefeed_auto_stop_1.usertable not exists for 2-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 table processor_delay.t40 not exists for 2-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 35 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test {"level":"warn","ts":"2025-04-04T15:48:16.024231+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002014c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:48:16.031925+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ebe8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:48:16.133051+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000237dc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table move_table.usertable not exists for 2-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 21-th time, retry later wait process cdc.test exit for 2-th time... table processor_delay.t40 not exists for 3-th check, retry later table changefeed_auto_stop_1.usertable not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table move_table.usertable exists go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 check diff failed 22-th time, retry later go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 table processor_delay.t40 not exists for 4-th check, retry later table changefeed_auto_stop_1.usertable not exists for 4-th check, retry later check diff failed 23-th time, retry later table processor_delay.t40 not exists for 5-th check, retry later table changefeed_auto_stop_1.usertable not exists for 5-th check, retry later go: downloading github.com/ardielle/ardielle-go v1.5.2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 24-th time, retry later table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table processor_delay.t40 exists table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 25-th time, retry later table processor_delay.t42 not exists for 2-th check, retry later table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 26-th time, retry later table processor_delay.t42 not exists for 3-th check, retry later table changefeed_auto_stop_3.usertable not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361679261390754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1716bc0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:14397, start at 2025-04-04 15:48:26.200665215 +0800 CST m=+2.503829690 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:26.209 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:26.259 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:26.259 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361679261390754 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1716bc0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:14397, start at 2025-04-04 15:48:26.200665215 +0800 CST m=+2.503829690 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:26.209 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:26.259 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:26.259 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361677521827686 TiDB Cluster ID. tikv_gc_leader_uuid 657ff171864000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:14460, start at 2025-04-04 15:48:26.294400178 +0800 CST m=+2.518480879 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:26.303 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:26.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:26.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 27-th time, retry later table processor_delay.t42 not exists for 4-th check, retry later table changefeed_auto_stop_3.usertable not exists for 3-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.15701.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t42 not exists for 5-th check, retry later table changefeed_auto_stop_3.usertable not exists for 4-th check, retry later Verifying downstream PD is started... check diff failed 28-th time, retry later + set +x + tso='457114362958315521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114362958315521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:48:31 CST 2025] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1573915741.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table changefeed_auto_stop_3.usertable not exists for 5-th check, retry later check diff failed 29-th time, retry later table processor_delay.t42 exists table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later check diff failed 30-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:34 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6becb55-ffc1-4063-b655-b33c5db177d0 {"id":"e6becb55-ffc1-4063-b655-b33c5db177d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752912} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5a40ce3 e6becb55-ffc1-4063-b655-b33c5db177d0 /tidb/cdc/default/default/upstream/7489361679261390754 {"id":7489361679261390754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6becb55-ffc1-4063-b655-b33c5db177d0 {"id":"e6becb55-ffc1-4063-b655-b33c5db177d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752912} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5a40ce3 e6becb55-ffc1-4063-b655-b33c5db177d0 /tidb/cdc/default/default/upstream/7489361679261390754 {"id":7489361679261390754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6becb55-ffc1-4063-b655-b33c5db177d0 {"id":"e6becb55-ffc1-4063-b655-b33c5db177d0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752912} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5a40ce3 e6becb55-ffc1-4063-b655-b33c5db177d0 /tidb/cdc/default/default/upstream/7489361679261390754 {"id":7489361679261390754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.15794.out cli changefeed create --start-ts=457114362958315521 '--sink-uri=file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-14579?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 2a0d9a97-5542-4431-8cf3-c249df060a88 Info: {"upstream_id":7489361679261390754,"namespace":"default","id":"2a0d9a97-5542-4431-8cf3-c249df060a88","sink_uri":"file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-14579?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:48:35.380584228+08:00","start_ts":457114362958315521,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114362958315521,"checkpoint_ts":457114362958315521,"checkpoint_time":"2025-04-04 15:48:30.455"} PASS table processor_delay.t44 not exists for 2-th check, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/savepoint + sink_uri='file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-14579?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/savepoint ++ date + echo '[Fri Apr 4 15:48:36 CST 2025] <<<<<< START storage consumer in savepoint case >>>>>>' [Fri Apr 4 15:48:36 CST 2025] <<<<<< START storage consumer in savepoint case >>>>>> + cd /tmp/tidb_cdc_test/savepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/savepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/savepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-14579?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t44 not exists for 3-th check, retry later check diff failed 2-th time, retry later check diff failed at last An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_uk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log for full details An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_pk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log for full details A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 A total of 2 table have been compared and all are equal. You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed1/output/sync_diff.log' An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_pk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log for full details An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_uk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log for full details A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/0 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_pk` ... failure Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_uk` ... equivalent _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed2/output/sync_diff.log' An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_pk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log for full details An error occurred while initializing diff: from upstream: please make sure the filter is correct.: the target has no table to be compared. source-table is ``test`.`update_uk``, please check log info in /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log for full details A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' A total of 2 tables need to be compared Comparing the table structure of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_uk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/1 Comparing the table structure of `test`.`update_pk` ... equivalent Comparing the table data of `test`.`update_uk` ... Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [>------------------------------------------------------------] 0% 0/2 Comparing the table data of `test`.`update_uk` ... equivalent Comparing the table data of `test`.`update_pk` ... _____________________________________________________________________________ Progress [==============================>------------------------------] 50% 0/1 Comparing the table data of `test`.`update_pk` ... failure _____________________________________________________________________________ Progress [============================================================>] 100% 0/0 Progress [============================================================>] 100% 0/0 The data of `test`.`update_pk` is not equal The rest of tables are all equal. A total of 2 tables have been compared, 1 tables finished, 1 tables failed, 0 tables skipped. The patch file has been generated in '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/fix-on-tidb0/' You can view the comparison details through '/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff-changefeed3/output/sync_diff.log' cat: /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/sync_diff/output/sync_diff.log: No such file or directory check_sync_diff failed as expected for changefeed3 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_update_pk_nonclustered.cli.14810.out cli changefeed create --start-ts=457114322070667269 '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed4?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed4.toml -c changefeed4 Create changefeed successfully! ID: changefeed4 Info: {"upstream_id":7489360998406299861,"namespace":"default","id":"changefeed4","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed4?flush-interval=5s","create_time":"2025-04-04T15:48:37.685125228+08:00","start_ts":457114322070667269,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":true,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"output_raw_change_event":false},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114322070667269,"checkpoint_ts":457114322070667269,"checkpoint_time":"2025-04-04 15:45:54.481"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table savepoint.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361723536339273 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17c734000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:12684, start at 2025-04-04 15:48:37.496098284 +0800 CST m=+2.632477105 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:37.505 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:37.503 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:37.503 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361723536339273 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17c734000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:12684, start at 2025-04-04 15:48:37.496098284 +0800 CST m=+2.632477105 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:37.505 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:37.503 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:37.503 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361728200776518 TiDB Cluster ID. tikv_gc_leader_uuid 657ff17c7f8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:12758, start at 2025-04-04 15:48:37.520509398 +0800 CST m=+2.556021765 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:37.564 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:37.552 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:37.552 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed4?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed4.toml + log_suffix=changefeed4 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered ++ date + echo '[Fri Apr 4 15:48:39 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>>' [Fri Apr 4 15:48:39 CST 2025] <<<<<< START storage consumer in csv_storage_update_pk_nonclustered case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed4.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/cdc_storage_consumerchangefeed4.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_update_pk_nonclustered/storage_test/changefeed4?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/conf/changefeed4.toml Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t44 not exists for 4-th check, retry later check diff failed 3-th time, retry later table savepoint.finish_mark not exists for 2-th check, retry later table processor_delay.t44 not exists for 5-th check, retry later + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.14116.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 4-th time, retry later table savepoint.finish_mark not exists for 3-th check, retry later + set +x + tso='457114365880696833 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114365880696833 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114365880696833 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:48:43 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1414914151.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:48:41.050 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:48:41.054 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:48:41.054 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=2a24660d-444f-4ade-854a-f72fd2e294e2] [2025/04/04 15:48:41.055 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:48:41.055 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:48:41.178 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=2a24660d-444f-4ade-854a-f72fd2e294e2] [processorDetail="{\"table_ids\":[114,116]}"] [2025/04/04 15:48:41.327 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=677a764e-135c-489f-95c4-aa6441131043] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:48:41.478 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=f024d26e-1a51-4645-b7fd-40c0849e0bbd] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:48:41.478 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"2a24660d-444f-4ade-854a-f72fd2e294e2\":[{\"ID\":114,\"Changefeed\":\"05196a3b-b86d-44be-a5d8-68612b795c06\"},{\"ID\":116,\"Changefeed\":\"05196a3b-b86d-44be-a5d8-68612b795c06\"}],\"677a764e-135c-489f-95c4-aa6441131043\":[],\"f024d26e-1a51-4645-b7fd-40c0849e0bbd\":[]}"] [2025/04/04 15:48:41.478 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=05196a3b-b86d-44be-a5d8-68612b795c06&target-cp-id=f024d26e-1a51-4645-b7fd-40c0849e0bbd&table-id=114"] [2025/04/04 15:48:41.528 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=114] [2025/04/04 15:48:41.528 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=05196a3b-b86d-44be-a5d8-68612b795c06&target-cp-id=f024d26e-1a51-4645-b7fd-40c0849e0bbd&table-id=116"] [2025/04/04 15:48:41.577 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:48:41.577 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=2a24660d-444f-4ade-854a-f72fd2e294e2] [targetCapture=f024d26e-1a51-4645-b7fd-40c0849e0bbd] table move_table.check1 exists check diff successfully table processor_delay.t44 exists table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table savepoint.finish_mark not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:45 CST 2025] <<<<<< run test case changefeed_auto_stop success! >>>>>> table processor_delay.t46 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:46 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f3f0207b-6fb6-4f98-80c7-6644e15dfbcb {"id":"f3f0207b-6fb6-4f98-80c7-6644e15dfbcb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752923} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5cf98e8 f3f0207b-6fb6-4f98-80c7-6644e15dfbcb /tidb/cdc/default/default/upstream/7489361723536339273 {"id":7489361723536339273,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f3f0207b-6fb6-4f98-80c7-6644e15dfbcb {"id":"f3f0207b-6fb6-4f98-80c7-6644e15dfbcb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752923} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5cf98e8 f3f0207b-6fb6-4f98-80c7-6644e15dfbcb /tidb/cdc/default/default/upstream/7489361723536339273 {"id":7489361723536339273,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f3f0207b-6fb6-4f98-80c7-6644e15dfbcb {"id":"f3f0207b-6fb6-4f98-80c7-6644e15dfbcb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752923} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc5cf98e8 f3f0207b-6fb6-4f98-80c7-6644e15dfbcb /tidb/cdc/default/default/upstream/7489361723536339273 {"id":7489361723536339273,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114365880696833 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.14206.out cli changefeed create --start-ts=457114365880696833 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml [2025/04/04 15:48:46.644 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:48:46.646 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:48:46.647 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=2a24660d-444f-4ade-854a-f72fd2e294e2] [2025/04/04 15:48:46.647 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:48:46.647 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361723536339273,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:48:46.575172429+08:00","start_ts":457114365880696833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114365880696833,"checkpoint_ts":457114365880696833,"checkpoint_time":"2025-04-04 15:48:41.603"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:48:46.777 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=2a24660d-444f-4ade-854a-f72fd2e294e2] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:48:46.928 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=677a764e-135c-489f-95c4-aa6441131043] [processorDetail="{\"table_ids\":[118]}"] table savepoint.finish_mark not exists for 5-th check, retry later [2025/04/04 15:48:47.078 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=05196a3b-b86d-44be-a5d8-68612b795c06] [captureID=f024d26e-1a51-4645-b7fd-40c0849e0bbd] [processorDetail="{\"table_ids\":[116]}"] [2025/04/04 15:48:47.078 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"2a24660d-444f-4ade-854a-f72fd2e294e2\":[],\"677a764e-135c-489f-95c4-aa6441131043\":[{\"ID\":118,\"Changefeed\":\"05196a3b-b86d-44be-a5d8-68612b795c06\"}],\"f024d26e-1a51-4645-b7fd-40c0849e0bbd\":[{\"ID\":116,\"Changefeed\":\"05196a3b-b86d-44be-a5d8-68612b795c06\"}]}"] [2025/04/04 15:48:47.078 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=05196a3b-b86d-44be-a5d8-68612b795c06&target-cp-id=f024d26e-1a51-4645-b7fd-40c0849e0bbd&table-id=118"] [2025/04/04 15:48:47.127 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=118] [2025/04/04 15:48:47.127 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=677a764e-135c-489f-95c4-aa6441131043] [targetCapture=f024d26e-1a51-4645-b7fd-40c0849e0bbd] table processor_delay.t46 not exists for 3-th check, retry later check diff failed 1-th time, retry later + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table savepoint.finish_mark exists check diff successfully check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... check diff failed 2-th time, retry later table processor_delay.t46 not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... check diff successfully table move_table.check2 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:48:50 CST 2025] <<<<<< run test case savepoint success! >>>>>> table test.t1 exists + sleep 5 check diff successfully [Fri Apr 4 15:48:51 CST 2025] <<<<<< run test case csv_storage_update_pk_nonclustered success! >>>>>> table processor_delay.t46 not exists for 5-th check, retry later table move_table.check2 not exists for 2-th check, retry later table processor_delay.t46 exists table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later table move_table.check2 not exists for 3-th check, retry later table processor_delay.t48 not exists for 2-th check, retry later + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 12150 24.6 0.6 4851700 2496460 ? Sl 15:48 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 12151 23.7 0.6 4845044 2492488 ? Sl 15:48 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 12152 34.5 0.6 4883444 2541540 ? Sl 15:48 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 12154 33.1 0.6 4872184 2526816 ? Sl 15:48 0:07 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 6394 0 --:--:-- --:--:-- --:--:-- 6567 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:48:53.903","puller_resolved_ts":"2025-04-04 15:48:48.054","last_synced_ts":"2025-04-04 15:48:48.103","now_ts":"2025-04-04 15:48:55.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:53.903","puller_resolved_ts":"2025-04-04' '15:48:48.054","last_synced_ts":"2025-04-04' '15:48:48.103","now_ts":"2025-04-04' '15:48:55.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:53.903","puller_resolved_ts":"2025-04-04' '15:48:48.054","last_synced_ts":"2025-04-04' '15:48:48.103","now_ts":"2025-04-04' '15:48:55.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table move_table.check2 not exists for 4-th check, retry later table processor_delay.t48 not exists for 3-th check, retry later table move_table.check2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table processor_delay.t48 not exists for 4-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:49:00 CST 2025] <<<<<< run test case move_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t48 not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... table processor_delay.t48 exists table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later table processor_delay.t50 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t50 not exists for 3-th check, retry later table processor_delay.t50 not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + '[' storage == mysql ']' + stop_tidb_cluster Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t50 not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t50 exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + stop_tidb_cluster start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... wait process cdc.test exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361885563399182 TiDB Cluster ID. tikv_gc_leader_uuid 657ff19fef00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:16977, start at 2025-04-04 15:49:13.791740435 +0800 CST m=+2.524237963 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:13.801 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:13.788 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:13.788 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361885563399182 TiDB Cluster ID. tikv_gc_leader_uuid 657ff19fef00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:16977, start at 2025-04-04 15:49:13.791740435 +0800 CST m=+2.524237963 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:13.801 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:13.788 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:13.788 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361882385113331 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a00a40005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:17056, start at 2025-04-04 15:49:13.903499529 +0800 CST m=+2.530607087 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:13.911 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:13.897 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:13.897 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:49:16 CST 2025] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.18297.out cli tso query --pd=http://127.0.0.1:2379 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage + set +x + tso='457114375445020674 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114375445020674 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114375445020674 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:49:19 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1832918331.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:22 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63d405eb-e327-4f85-bcfb-46ce6efaf9e8 {"id":"63d405eb-e327-4f85-bcfb-46ce6efaf9e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752959} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc65fe1e4 63d405eb-e327-4f85-bcfb-46ce6efaf9e8 /tidb/cdc/default/default/upstream/7489361885563399182 {"id":7489361885563399182,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63d405eb-e327-4f85-bcfb-46ce6efaf9e8 {"id":"63d405eb-e327-4f85-bcfb-46ce6efaf9e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752959} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc65fe1e4 63d405eb-e327-4f85-bcfb-46ce6efaf9e8 /tidb/cdc/default/default/upstream/7489361885563399182 {"id":7489361885563399182,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63d405eb-e327-4f85-bcfb-46ce6efaf9e8 {"id":"63d405eb-e327-4f85-bcfb-46ce6efaf9e8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752959} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc65fe1e4 63d405eb-e327-4f85-bcfb-46ce6efaf9e8 /tidb/cdc/default/default/upstream/7489361885563399182 {"id":7489361885563399182,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114375445020674 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.18381.out cli changefeed create --start-ts=457114375445020674 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361885563399182,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:49:23.089809923+08:00","start_ts":457114375445020674,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114375445020674,"checkpoint_ts":457114375445020674,"checkpoint_time":"2025-04-04 15:49:18.088"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 5261 0 --:--:-- --:--:-- --:--:-- 5261 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:49:24.288","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:49:24.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:49:24.288","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:49:24.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:49:24.288","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:49:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:49:24.288' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:49:24.288","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:49:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:49:24.288","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:49:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:49:24' + echo 'sink_checkpoint_ts is 2025-04-04' 15:49:24.288 sink_checkpoint_ts is 2025-04-04 15:49:24.288 ++ date -d '2025-04-04 15:49:24.288' +%s + checkpoint_timestamp=1743752964 ++ date -d '2025-04-04 15:49:24' +%s + current_timestamp=1743752964 + '[' 0 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361924739468774 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a9150000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:30684, start at 2025-04-04 15:49:23.203913665 +0800 CST m=+2.207859572 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:23.212 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:23.206 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:23.206 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361924739468774 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a9150000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:30684, start at 2025-04-04 15:49:23.203913665 +0800 CST m=+2.207859572 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:23.212 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:23.206 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:23.206 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361928811084444 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a93a4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:30747, start at 2025-04-04 15:49:23.34559142 +0800 CST m=+2.251708957 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:23.353 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:23.354 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:23.354 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t1 exists + sleep 5 [Fri Apr 4 15:49:27 CST 2025] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.3204332045.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:30 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/25c19b8b-fe4d-4adc-9212-6080047e97d4 {"id":"25c19b8b-fe4d-4adc-9212-6080047e97d4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752967} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc687c9d5 25c19b8b-fe4d-4adc-9212-6080047e97d4 /tidb/cdc/default/default/upstream/7489361924739468774 {"id":7489361924739468774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/25c19b8b-fe4d-4adc-9212-6080047e97d4 {"id":"25c19b8b-fe4d-4adc-9212-6080047e97d4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752967} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc687c9d5 25c19b8b-fe4d-4adc-9212-6080047e97d4 /tidb/cdc/default/default/upstream/7489361924739468774 {"id":7489361924739468774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/25c19b8b-fe4d-4adc-9212-6080047e97d4 {"id":"25c19b8b-fe4d-4adc-9212-6080047e97d4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752967} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc687c9d5 25c19b8b-fe4d-4adc-9212-6080047e97d4 /tidb/cdc/default/default/upstream/7489361924739468774 {"id":7489361924739468774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 790b8cc7-8135-4554-a845-93644266ab5b Info: {"upstream_id":7489361924739468774,"namespace":"default","id":"790b8cc7-8135-4554-a845-93644266ab5b","sink_uri":"file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-5953?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:49:30.673012201+08:00","start_ts":457114378713432069,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114378713432069,"checkpoint_ts":457114378713432069,"checkpoint_time":"2025-04-04 15:49:30.556"} + workdir=/tmp/tidb_cdc_test/autorandom + sink_uri='file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-5953?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/autorandom ++ date + echo '[Fri Apr 4 15:49:30 CST 2025] <<<<<< START storage consumer in autorandom case >>>>>>' [Fri Apr 4 15:49:30 CST 2025] <<<<<< START storage consumer in autorandom case >>>>>> + cd /tmp/tidb_cdc_test/autorandom + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/autorandom + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/autorandom/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-5953?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 9346 0 --:--:-- --:--:-- --:--:-- 9346 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:49:31.288","puller_resolved_ts":"2025-04-04 15:49:24.639","last_synced_ts":"2025-04-04 15:49:24.639","now_ts":"2025-04-04 15:49:31.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:49:31.288","puller_resolved_ts":"2025-04-04' '15:49:24.639","last_synced_ts":"2025-04-04' '15:49:24.639","now_ts":"2025-04-04' '15:49:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:49:31.288","puller_resolved_ts":"2025-04-04' '15:49:24.639","last_synced_ts":"2025-04-04' '15:49:24.639","now_ts":"2025-04-04' '15:49:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table autorandom_test.table_a not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table autorandom_test.table_a not exists for 2-th check, retry later table autorandom_test.table_a not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table autorandom_test.table_a not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table autorandom_test.table_a not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table autorandom_test.table_a exists check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362000986138676 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1baa28000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:8111, start at 2025-04-04 15:49:41.137675649 +0800 CST m=+2.223637524 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:41.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:41.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:41.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362000986138676 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1baa28000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:8111, start at 2025-04-04 15:49:41.137675649 +0800 CST m=+2.223637524 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:41.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:41.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:41.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362003417552525 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1ba9e40008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-5q5-w4lvz, pid:8167, start at 2025-04-04 15:49:41.126422683 +0800 CST m=+2.152881329 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:41.134 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:41.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:41.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 2-th time... Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:49:43 CST 2025] <<<<<< run test case autorandom success! >>>>>> [Fri Apr 4 15:49:45 CST 2025] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.94109412.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:48 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5460fca7-767f-440f-beb8-bbd0c92bb336 {"id":"5460fca7-767f-440f-beb8-bbd0c92bb336","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752985} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6cbdbd4 5460fca7-767f-440f-beb8-bbd0c92bb336 /tidb/cdc/default/default/upstream/7489362000986138676 {"id":7489362000986138676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5460fca7-767f-440f-beb8-bbd0c92bb336 {"id":"5460fca7-767f-440f-beb8-bbd0c92bb336","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752985} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6cbdbd4 5460fca7-767f-440f-beb8-bbd0c92bb336 /tidb/cdc/default/default/upstream/7489362000986138676 {"id":7489362000986138676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5460fca7-767f-440f-beb8-bbd0c92bb336 {"id":"5460fca7-767f-440f-beb8-bbd0c92bb336","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743752985} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc6cbdbd4 5460fca7-767f-440f-beb8-bbd0c92bb336 /tidb/cdc/default/default/upstream/7489362000986138676 {"id":7489362000986138676,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/sink_hang + sink_uri='file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-29784?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_hang ++ date + echo '[Fri Apr 4 15:49:48 CST 2025] <<<<<< START storage consumer in sink_hang case >>>>>>' [Fri Apr 4 15:49:48 CST 2025] <<<<<< START storage consumer in sink_hang case >>>>>> + cd /tmp/tidb_cdc_test/sink_hang + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_hang + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_hang/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-29784?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_status 127.0.0.1:8300 a21ab8f3-aaaf-46f0-a381-628571d5ae9d normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=a21ab8f3-aaaf-46f0-a381-628571d5ae9d + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/a21ab8f3-aaaf-46f0-a381-628571d5ae9d/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 614 0 --:--:-- --:--:-- --:--:-- 614 + info='{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' + echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' {"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066} ++ echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 a21ab8f3-aaaf-46f0-a381-628571d5ae9d normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=a21ab8f3-aaaf-46f0-a381-628571d5ae9d + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/a21ab8f3-aaaf-46f0-a381-628571d5ae9d/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 682 0 --:--:-- --:--:-- --:--:-- 682 + info='{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' + echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' {"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066} ++ echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":457114383595864066,"checkpoint_ts":457114383595864066}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later table sink_hang.t1 not exists for 2-th check, retry later table sink_hang.t1 not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table sink_hang.t1 not exists for 4-th check, retry later table sink_hang.t1 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table sink_hang.t1 exists table sink_hang.t2 exists check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362111232705815 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d23ec000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:33265, start at 2025-04-04 15:50:05.34254435 +0800 CST m=+2.152636565 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:05.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:05.357 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:05.357 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362111232705815 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d23ec000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:33265, start at 2025-04-04 15:50:05.34254435 +0800 CST m=+2.152636565 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:05.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:05.357 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:05.357 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362110690210268 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1d257c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-42j-21qtt, pid:33334, start at 2025-04-04 15:50:05.437910492 +0800 CST m=+2.180247585 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:05.446 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:05.407 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:05.407 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.34589.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:10 CST 2025] <<<<<< run test case sink_hang success! >>>>>> + set +x + tso='457114389029060610 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114389029060610 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:50:11 CST 2025] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.3462734629.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:50:14 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 {"id":"4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753011} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7308be3 4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 /tidb/cdc/default/default/upstream/7489362111232705815 {"id":7489362111232705815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 {"id":"4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753011} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7308be3 4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 /tidb/cdc/default/default/upstream/7489362111232705815 {"id":7489362111232705815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 {"id":"4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753011} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7308be3 4bf2811f-b327-4ad8-a19f-eeba1f3a4cd4 /tidb/cdc/default/default/upstream/7489362111232705815 {"id":7489362111232705815,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.34682.out cli changefeed create --start-ts=457114389029060610 '--sink-uri=file:///tmp/tidb_cdc_test/generate_column/storage_test/ticdc-generate-column-test-27086?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 8f3d873c-72e5-4865-92c6-70e6896dfdc5 Info: {"upstream_id":7489362111232705815,"namespace":"default","id":"8f3d873c-72e5-4865-92c6-70e6896dfdc5","sink_uri":"file:///tmp/tidb_cdc_test/generate_column/storage_test/ticdc-generate-column-test-27086?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:50:14.809817082+08:00","start_ts":457114389029060610,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114389029060610,"checkpoint_ts":457114389029060610,"checkpoint_time":"2025-04-04 15:50:09.907"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + workdir=/tmp/tidb_cdc_test/generate_column + sink_uri='file:///tmp/tidb_cdc_test/generate_column/storage_test/ticdc-generate-column-test-27086?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/generate_column ++ date + echo '[Fri Apr 4 15:50:16 CST 2025] <<<<<< START storage consumer in generate_column case >>>>>>' [Fri Apr 4 15:50:16 CST 2025] <<<<<< START storage consumer in generate_column case >>>>>> + cd /tmp/tidb_cdc_test/generate_column + '[' /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/generate_column + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/generate_column/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/generate_column/storage_test/ticdc-generate-column-test-27086?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/conf/changefeed.toml table generate_column.t1 not exists for 1-th check, retry later table generate_column.t1 not exists for 2-th check, retry later table generate_column.t1 not exists for 3-th check, retry later table generate_column.t1 not exists for 4-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table generate_column.t1 not exists for 5-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table generate_column.t1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:50:29 CST 2025] <<<<<< run test case generate_column success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 25821 0 --:--:-- --:--:-- --:--:-- 25821 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:48:54.904","puller_resolved_ts":"2025-04-04 15:48:54.904","last_synced_ts":"2025-04-04 15:48:48.103","now_ts":"2025-04-04 15:51:05.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:54.904","puller_resolved_ts":"2025-04-04' '15:48:54.904","last_synced_ts":"2025-04-04' '15:48:48.103","now_ts":"2025-04-04' '15:51:05.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:54.904","puller_resolved_ts":"2025-04-04' '15:48:54.904","last_synced_ts":"2025-04-04' '15:48:48.103","now_ts":"2025-04-04' '15:51:05.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362456729534044 TiDB Cluster ID. tikv_gc_leader_uuid 657ff221b7c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:15352, start at 2025-04-04 15:51:26.703981349 +0800 CST m=+2.494665300 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:26.712 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:26.759 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:26.759 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362456729534044 TiDB Cluster ID. tikv_gc_leader_uuid 657ff221b7c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:15352, start at 2025-04-04 15:51:26.703981349 +0800 CST m=+2.494665300 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:26.712 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:26.759 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:26.759 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362456849098949 TiDB Cluster ID. tikv_gc_leader_uuid 657ff221c5c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:15427, start at 2025-04-04 15:51:26.783224059 +0800 CST m=+2.491636372 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:26.791 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:26.794 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:26.794 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16722.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114410270588930 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114410270588930 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114410270588930 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:51:32 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1675616758.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:35 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/616df782-fe9f-474a-8b94-77976ca760cd {"id":"616df782-fe9f-474a-8b94-77976ca760cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753092} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc86734e0 616df782-fe9f-474a-8b94-77976ca760cd /tidb/cdc/default/default/upstream/7489362456729534044 {"id":7489362456729534044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/616df782-fe9f-474a-8b94-77976ca760cd {"id":"616df782-fe9f-474a-8b94-77976ca760cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753092} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc86734e0 616df782-fe9f-474a-8b94-77976ca760cd /tidb/cdc/default/default/upstream/7489362456729534044 {"id":7489362456729534044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/616df782-fe9f-474a-8b94-77976ca760cd {"id":"616df782-fe9f-474a-8b94-77976ca760cd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753092} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc86734e0 616df782-fe9f-474a-8b94-77976ca760cd /tidb/cdc/default/default/upstream/7489362456729534044 {"id":7489362456729534044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114410270588930 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16809.out cli changefeed create --start-ts=457114410270588930 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362456729534044,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:51:35.923389715+08:00","start_ts":457114410270588930,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114410270588930,"checkpoint_ts":457114410270588930,"checkpoint_time":"2025-04-04 15:51:30.937"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 15352 10.4 0.0 3110104 216108 ? Sl 15:51 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753084198453443.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 15361 4.2 0.0 3479212 187096 ? Sl 15:51 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753084206383910.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 15427 11.1 0.0 3258688 219172 ? Sl 15:51 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743753084280226170.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2531 0 --:--:-- --:--:-- --:--:-- 2531 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:51:43.287","puller_resolved_ts":"2025-04-04 15:51:37.387","last_synced_ts":"2025-04-04 15:51:37.387","now_ts":"2025-04-04 15:51:44.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:43.287","puller_resolved_ts":"2025-04-04' '15:51:37.387","last_synced_ts":"2025-04-04' '15:51:37.387","now_ts":"2025-04-04' '15:51:44.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:43.287","puller_resolved_ts":"2025-04-04' '15:51:37.387","last_synced_ts":"2025-04-04' '15:51:37.387","now_ts":"2025-04-04' '15:51:44.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:51:41.488","puller_resolved_ts":"2025-04-04 15:51:33.488","last_synced_ts":"2025-04-04 15:49:24.639","now_ts":"2025-04-04 15:51:41.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:51:41.488","puller_resolved_ts":"2025-04-04' '15:51:33.488","last_synced_ts":"2025-04-04' '15:49:24.639","now_ts":"2025-04-04' '15:51:41.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 16180 9.7 0.0 15484896 150988 ? Sl 15:49 0:15 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 16244 6.9 0.0 15410652 144060 ? Sl 15:49 0:10 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1743753108.8804674,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005543880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753108.8805447,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743753108.9130204,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0011ca8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753108.9130948,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743753108.9695983,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004a79500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743753108.969684,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:51:56.375083+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:51:56.380869+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:51:56.481319+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":1743753121.961876,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005543880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743753122.0546,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004a79500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:02.375978+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:02.382223+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:02.483082+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:52:08.377318+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:08.382581+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:08.483838+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:52:14.378241+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:14.383512+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:14.485136+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:52:16.365824+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:52:16.365883+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:52:16.369392+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:52:16.369434+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:52:16.470277+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:52:16.470319+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:52:20.379289+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:20.384967+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:20.485696+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1743753143.8822346,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005543880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753143.8822722,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743753143.9143796,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0011ca8c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743753143.9144282,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743753143.97147,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004a79500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743753143.9715211,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:52:26.381215+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:26.386606+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:26.48682+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 35 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... {"level":"warn","ts":"2025-04-04T15:52:32.382211+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f84700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:32.387757+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:52:32.487646+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00080c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362816828196233 TiDB Cluster ID. tikv_gc_leader_uuid 657ff274198000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:19523, start at 2025-04-04 15:52:51.095352382 +0800 CST m=+2.422696993 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:51.102 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:51.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:51.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362816828196233 TiDB Cluster ID. tikv_gc_leader_uuid 657ff274198000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:19523, start at 2025-04-04 15:52:51.095352382 +0800 CST m=+2.422696993 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:51.102 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:51.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:51.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362817694311314 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2743280004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:19579, start at 2025-04-04 15:52:51.164352615 +0800 CST m=+2.432480689 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:51.173 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:51.146 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:51.146 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.20949.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114432424116225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114432424116225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114432424116225 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:52:56 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2098420986.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:52:59 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a09022b7-d994-4f57-b3bf-f81f421b3314 {"id":"a09022b7-d994-4f57-b3bf-f81f421b3314","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753177} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc9b6fde3 a09022b7-d994-4f57-b3bf-f81f421b3314 /tidb/cdc/default/default/upstream/7489362816828196233 {"id":7489362816828196233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a09022b7-d994-4f57-b3bf-f81f421b3314 {"id":"a09022b7-d994-4f57-b3bf-f81f421b3314","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753177} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc9b6fde3 a09022b7-d994-4f57-b3bf-f81f421b3314 /tidb/cdc/default/default/upstream/7489362816828196233 {"id":7489362816828196233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a09022b7-d994-4f57-b3bf-f81f421b3314 {"id":"a09022b7-d994-4f57-b3bf-f81f421b3314","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753177} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc9b6fde3 a09022b7-d994-4f57-b3bf-f81f421b3314 /tidb/cdc/default/default/upstream/7489362816828196233 {"id":7489362816828196233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114432424116225 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21040.out cli changefeed create --start-ts=457114432424116225 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362816828196233,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:53:00.391452752+08:00","start_ts":457114432424116225,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114432424116225,"checkpoint_ts":457114432424116225,"checkpoint_time":"2025-04-04 15:52:55.446"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 18982 31.6 0.6 4892664 2539900 ? Sl 15:52 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 18983 22.5 0.6 4854256 2497532 ? Sl 15:52 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 18984 22.2 0.6 4851700 2469372 ? Sl 15:52 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 18986 30.4 0.6 4881396 2527756 ? Sl 15:52 0:07 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 9720 0 --:--:-- --:--:-- --:--:-- 9720 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:53:08.696","puller_resolved_ts":"2025-04-04 15:53:01.847","last_synced_ts":"2025-04-04 15:53:02.296","now_ts":"2025-04-04 15:53:09.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:08.696","puller_resolved_ts":"2025-04-04' '15:53:01.847","last_synced_ts":"2025-04-04' '15:53:02.296","now_ts":"2025-04-04' '15:53:09.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:08.696","puller_resolved_ts":"2025-04-04' '15:53:01.847","last_synced_ts":"2025-04-04' '15:53:02.296","now_ts":"2025-04-04' '15:53:09.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6906 0 --:--:-- --:--:-- --:--:-- 6906 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:53:53.437","puller_resolved_ts":"2025-04-04 15:53:46.437","last_synced_ts":"2025-04-04 15:51:37.387","now_ts":"2025-04-04 15:53:54.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:53:53.437","puller_resolved_ts":"2025-04-04' '15:53:46.437","last_synced_ts":"2025-04-04' '15:51:37.387","now_ts":"2025-04-04' '15:53:54.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:53:53.437","puller_resolved_ts":"2025-04-04' '15:53:46.437","last_synced_ts":"2025-04-04' '15:51:37.387","now_ts":"2025-04-04' '15:53:54.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363179951772716 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2c737c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:18031, start at 2025-04-04 15:54:16.20461259 +0800 CST m=+2.235163190 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:56:16.213 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:54:16.255 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:44:16.255 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363179951772716 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2c737c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:18031, start at 2025-04-04 15:54:16.20461259 +0800 CST m=+2.235163190 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:56:16.213 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:54:16.255 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:44:16.255 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363181605998006 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2c7684000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-965-hrj58, pid:18095, start at 2025-04-04 15:54:16.38374199 +0800 CST m=+2.319679744 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:56:16.391 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:54:16.353 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:44:16.353 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19324.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114454774251521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114454774251521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114454774251521 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:54:22 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1936119363.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:54:25 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e353fab-6fb0-43ea-9a76-40196fa6faf8 {"id":"4e353fab-6fb0-43ea-9a76-40196fa6faf8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753262} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcafc38eb 4e353fab-6fb0-43ea-9a76-40196fa6faf8 /tidb/cdc/default/default/upstream/7489363179951772716 {"id":7489363179951772716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e353fab-6fb0-43ea-9a76-40196fa6faf8 {"id":"4e353fab-6fb0-43ea-9a76-40196fa6faf8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753262} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcafc38eb 4e353fab-6fb0-43ea-9a76-40196fa6faf8 /tidb/cdc/default/default/upstream/7489363179951772716 {"id":7489363179951772716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e353fab-6fb0-43ea-9a76-40196fa6faf8 {"id":"4e353fab-6fb0-43ea-9a76-40196fa6faf8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753262} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcafc38eb 4e353fab-6fb0-43ea-9a76-40196fa6faf8 /tidb/cdc/default/default/upstream/7489363179951772716 {"id":7489363179951772716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114454774251521 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19421.out cli changefeed create --start-ts=457114454774251521 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489363179951772716,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:54:25.645060533+08:00","start_ts":457114454774251521,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114454774251521,"checkpoint_ts":457114454774251521,"checkpoint_time":"2025-04-04 15:54:20.705"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 14755 0 --:--:-- --:--:-- --:--:-- 14755 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:54:20.705","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:54:47.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:54:20.705","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:54:47.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:54:20.705","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:54:47.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Fri Apr 4 15:54:58 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Fri Apr 4 15:54:58 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 24100 0 --:--:-- --:--:-- --:--:-- 24100 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:53:08.696","puller_resolved_ts":"2025-04-04 15:53:08.696","last_synced_ts":"2025-04-04 15:53:02.296","now_ts":"2025-04-04 15:55:19.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:08.696","puller_resolved_ts":"2025-04-04' '15:53:08.696","last_synced_ts":"2025-04-04' '15:53:02.296","now_ts":"2025-04-04' '15:55:19.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:08.696","puller_resolved_ts":"2025-04-04' '15:53:08.696","last_synced_ts":"2025-04-04' '15:53:02.296","now_ts":"2025-04-04' '15:55:19.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363542698323097 TiDB Cluster ID. tikv_gc_leader_uuid 657ff318af4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:22284, start at 2025-04-04 15:55:39.615134216 +0800 CST m=+2.162917204 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:57:39.664 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:55:39.659 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:45:39.659 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363542698323097 TiDB Cluster ID. tikv_gc_leader_uuid 657ff318af4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:22284, start at 2025-04-04 15:55:39.615134216 +0800 CST m=+2.162917204 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:57:39.664 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:55:39.659 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:45:39.659 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363542932604200 TiDB Cluster ID. tikv_gc_leader_uuid 657ff318c840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:22350, start at 2025-04-04 15:55:39.682480459 +0800 CST m=+2.172138728 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:57:39.690 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:55:39.681 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:45:39.681 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.23569.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114476643876865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114476643876865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114476643876865 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:55:45 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2359623598.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:55:48 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 {"id":"4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753345} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcc4470e4 4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 /tidb/cdc/default/default/upstream/7489363542698323097 {"id":7489363542698323097,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 {"id":"4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753345} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcc4470e4 4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 /tidb/cdc/default/default/upstream/7489363542698323097 {"id":7489363542698323097,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 {"id":"4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753345} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffcc4470e4 4f9b048e-bbdd-4927-bfb2-c3c0bf4a30a2 /tidb/cdc/default/default/upstream/7489363542698323097 {"id":7489363542698323097,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114476643876865 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.23653.out cli changefeed create --start-ts=457114476643876865 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489363542698323097,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:55:49.019235255+08:00","start_ts":457114476643876865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114476643876865,"checkpoint_ts":457114476643876865,"checkpoint_time":"2025-04-04 15:55:44.131"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 22275 4.4 0.0 3257248 185984 ? Sl 15:55 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753337434561351.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 22284 10.8 0.0 3332324 215644 ? Sl 15:55 0:01 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753337442266501.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 22350 11.1 0.0 3332228 216376 ? Sl 15:55 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753337499612578.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 5400 0 --:--:-- --:--:-- --:--:-- 5400 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:55:55.482","puller_resolved_ts":"2025-04-04 15:55:50.481","last_synced_ts":"2025-04-04 15:55:50.931","now_ts":"2025-04-04 15:55:56.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:55:55.482","puller_resolved_ts":"2025-04-04' '15:55:50.481","last_synced_ts":"2025-04-04' '15:55:50.931","now_ts":"2025-04-04' '15:55:56.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:55:55.482","puller_resolved_ts":"2025-04-04' '15:55:50.481","last_synced_ts":"2025-04-04' '15:55:50.931","now_ts":"2025-04-04' '15:55:56.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6906 0 --:--:-- --:--:-- --:--:-- 6906 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:58:05.631","puller_resolved_ts":"2025-04-04 15:57:59.632","last_synced_ts":"2025-04-04 15:55:50.931","now_ts":"2025-04-04 15:58:06.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:58:05.631","puller_resolved_ts":"2025-04-04' '15:57:59.632","last_synced_ts":"2025-04-04' '15:55:50.931","now_ts":"2025-04-04' '15:58:06.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:58:05.631","puller_resolved_ts":"2025-04-04' '15:57:59.632","last_synced_ts":"2025-04-04' '15:55:50.931","now_ts":"2025-04-04' '15:58:06.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489364249649711276 TiDB Cluster ID. tikv_gc_leader_uuid 657ff3ba4e4003f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:24886, start at 2025-04-04 15:58:25.109385032 +0800 CST m=+2.125406475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-16:00:25.115 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:58:25.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:48:25.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489364249649711276 TiDB Cluster ID. tikv_gc_leader_uuid 657ff3ba4e4003f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:24886, start at 2025-04-04 15:58:25.109385032 +0800 CST m=+2.125406475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-16:00:25.115 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:58:25.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:48:25.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489364253029000973 TiDB Cluster ID. tikv_gc_leader_uuid 657ff3ba5900002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-storage-test-3706-33h-j7bt5, pid:24946, start at 2025-04-04 15:58:25.159897408 +0800 CST m=+2.084014069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-16:00:25.168 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:58:25.174 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:48:25.174 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26176.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114520041553923 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114520041553923 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114520041553923 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:58:31 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2620626208.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:58:34 GMT < Content-Length: 818 < Content-Type: text/plain; charset=utf-8 < { [818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/325738d9-899b-45f8-af7c-1bcc2fd70082 {"id":"325738d9-899b-45f8-af7c-1bcc2fd70082","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffceced3ee 325738d9-899b-45f8-af7c-1bcc2fd70082 /tidb/cdc/default/default/upstream/7489364249649711276 {"id":7489364249649711276,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/325738d9-899b-45f8-af7c-1bcc2fd70082 {"id":"325738d9-899b-45f8-af7c-1bcc2fd70082","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffceced3ee 325738d9-899b-45f8-af7c-1bcc2fd70082 /tidb/cdc/default/default/upstream/7489364249649711276 {"id":7489364249649711276,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/325738d9-899b-45f8-af7c-1bcc2fd70082 {"id":"325738d9-899b-45f8-af7c-1bcc2fd70082","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1743753511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffceced3ee 325738d9-899b-45f8-af7c-1bcc2fd70082 /tidb/cdc/default/default/upstream/7489364249649711276 {"id":7489364249649711276,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114520041553923 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26265.out cli changefeed create --start-ts=457114520041553923 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489364249649711276,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:58:34.582504535+08:00","start_ts":457114520041553923,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114520041553923,"checkpoint_ts":457114520041553923,"checkpoint_time":"2025-04-04 15:58:29.680"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 13903 0 --:--:-- --:--:-- --:--:-- 13903 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:58:29.680","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:58:56.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:58:29.680","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:58:56.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:58:29.680","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:58:56.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 4 15:59:06 CST 2025] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 4 15:59:06 CST 2025] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_storage_test-3706/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS